Hostname: page-component-848d4c4894-2xdlg Total loading time: 0 Render date: 2024-06-29T01:21:53.309Z Has data issue: false hasContentIssue false
  • English
  • Français

A Manual Actions Expressive System (MAES)

Published online by Cambridge University Press:  12 November 2013

Rajmil Fischman
Rajmil Fischman*
Affiliation:
Music, School of Humanities, Keele University, Keele, Staffordshire, ST5 5BG, UK E-mail: r.a.fischman@keele.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

This article describes a manual actions expressive system (MAES) which aims to enable music creation and performance using natural hand actions (e.g. hitting virtual objects, or shaking them). Gestures are fully programmable and result from tracking and analysing hand motion and finger bend, potentially allowing performers to concentrate on natural actions from our daily use of the hands (e.g. the physical movement associated with hitting and shaking). Work carried out focused on the development of an approach for the creation of gestures based on intuitive metaphors, their implementation as software for composition and performance, and their realisation within a musical composition through the choice of suitable mappings, sonic materials and processes.

Type
Articles
Information
Organised Sound , Volume 18 , Special Issue 3: Re-wiring Electronic Music , December 2013 , pp. 328 - 345
Copyright
Copyright © Cambridge University Press 2013 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bencina, R. 2006. P5 Glove Developments. http://www.simulus.org/p5glove (accessed on 24 December 2012).Google Scholar
Berger, M. 2010. The GRIP MAESTRO: Idiomatic Mappings of Emotive Gestures for Control of Live Electroacoustic Music. Proceedings of the Conference on New Interfaces for Musical Expression. Sydney. http://www.nime.org/proceedings/2010/nime2010_419.pdf (accessed on 14 January 2013).Google Scholar
Berthaut, F., Katayose, H., Wakama, H., Totani, N., Sato, Y. 2011. First Person Shooters as Collaborative Multiprocess Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 44–7. http://www.nime.org/proceedings/2011/nime2011_044.pdf (accessed on 8 March 13).Google Scholar
Beyer, G., Meier, M. 2011. Music Interfaces for Novice Users: Composing Music on a Public Display with Hand Gestures. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 507–10. http://www.nime2011.org/proceedings/papers/M23-Beyer.pdf (accessed on 20 January 2013).Google Scholar
Bongers, B. 2007. Electronic Musical Instruments: Experiences of a New Luthier. Leonardo Music Journal 17: 9–16.Google Scholar
Cadoz, C. 1988. Instrumental Gesture and Musical Composition. Proceedings of the 1988 International Computer Music Conference. Cologne/San Francisco: ICMA, 1–12.Google Scholar
Cadoz, C., Luciani, A., Florens, J. 1984. Responsive Input Devices and Sound Synthesis by Simulation of Instrumental Mechanisms: The Cordis System. Computer Music Journal 8(3): 60–73.Google Scholar
Cadoz, C., Ramstein, C. 1990. Capture, Representation and ‘Composition’ of the Instrumental Gesture. Proceedings of the 1990 International Computer Music Conference. Glasgow/San Francisco: ICMA, 53–56.Google Scholar
Charles, J.F. 2008. A Tutorial on Spectral Sound Processing Using Max/MSP and Jitter. Computer Music Journal 32(3): 87–102.Google Scholar
Choi, I., Bargar, R., Goudeseune, C. 1995. A Manifold Interface for a High Dimensional Control Space. Proceedings of the 1995 International Computer Music Conference. Banff/San Francisco: ICMA, 385–92.Google Scholar
Comajuncosas, J.M., Barrachina, A., O'Connell, J., Guaus, E. 2011. Nuvolet: 3D Gesture-Driven Collaborative Audio Mosaicing. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 252–4. http://www.nime2011.org/proceedings/papers/G11-Comajuncosas.pdf (accessed on 23 January 2013).Google Scholar
Cook, P., Leider, C. 2000. Making the Computer Sing: The SqueezeVox. Proceedings of the XIII Colloquium on Musical Informatics. L'Aquila.Google Scholar
De Poli, G., Piccialli, A., Roads, C. 1991. Representation of Musical Signals. Cambridge, MA: The MIT Press.Google Scholar
de Quay, Y., Skogstad, S. 2010a. Dance Jockey. http://www.youtube.com/watch?v=m1OffxIArrA (accessed on 19 December 2012).Google Scholar
de Quay, Y., Skogstad, S. 2010b. Dance Jockey v0.3. http://www.youtube.com/watch?v=HaD9MJzW59s (accessed on 19 December 2012).Google Scholar
de Quay, Y., Skogstad, S. 2011. Dance Jockey @ NIME. https://www.youtube.com/watch?v=PgE6YxhHgvM (accessed on 19 December 2012).Google Scholar
DuBois, R.L. 2012. Interviews with Matthew Ostrowski: The Glove. http://cycling74.com/2012/10/16/interviews-with-matthew-ostrowski (accessed on 25 February 2013).Google Scholar
Essl, G., O'Modhrain, S. 2006. An Enactive Approach to the Design of New Tangible Musical Instruments. Organised Sound 11(3): 285–296.Google Scholar
Fels, S., Hinton, G. 1998. Glove-TalkII: A Neural-Network Interface which Maps Gestures to Parallel Formant Speech Synthesiser Controls. IEEE Transactions on Neural Networks 9(1): 205–212. http://www.cs.toronto.edu/~hinton/absps/glovetalkii.pdf (accessed on 20 December 2012).Google Scholar
Fels, S., Gadd, A., Mulder, A. 2002. Mapping Transparency Through Metaphor: Towards More Expressive Musical Instruments. Organised Sound 7(2): 109–126.Google Scholar
Fischman, R. 1997. The Phase Vocoder: Theory and Practice. Organised Sound 2(2): 127–145.Google Scholar
Fischman, R. 2011. Back to the Parlour. Sonic Ideas – Ideas Sónicas 3(2): 53–66.Google Scholar
Fischman, R. 2012. Ruraq Maki (2012). https://vimeo.com/55093629 (accessed on 17 December 2012).Google Scholar
Fischman, R. 2013 (forthcoming). MAES, MAX patch and P5GloveRF.mxe external object.Google Scholar
Gadd, A., Fels, S. 2002. MetaMuse: Metaphors for Expressive Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, 2002. Dublin. http://hct.ece.ubc.ca/publications/pdf/gadd-fels-NIME02.pdf (accessed on 20 December 2012).Google Scholar
Gillian, N., Paradiso, J. 2012. Digito: A Fine-Grain Gesturally Controlled Virtual Musical Instrument. Proceedings of the International Conference on New Interfaces for Musical Expression, 2012. Ann Arbor. http://www.eecs.umich.edu/nime2012/Proceedings/papers/248_Final_Manuscript.pdf (accessed on 17 January 2013).Google Scholar
Goto, S. 1999. The Aesthetics and Technological Aspects of Virtual Musical Instruments: The Case of the Super-Polm MIDI Violin. Leonardo Music Journal 9: 115–120.Google Scholar
Goto, S. 2005. Virtual Musical Instruments: Technological Aspects and Interactive Performance Issues. HZ Journal 6. http://hz-journal.org/n6/goto.html (accessed on 24 December 2012).Google Scholar
Goudeseune, C., Garnett, G., Johnson, T. 2001. Resonant Processing of Instrumental Sound Controlled by Spatial Position. New Instruments and Musical Expression workshop, SIGCHI ‘01. Seattle. http://zx81.isl.uiuc.edu/camilleg/nime01.pdf (accessed on 14 December 2012).Google Scholar
Hunt, A.D., Paradis, M., Wanderley, M. 2003. The Importance of Parameter Mapping in Electronic Instrument Design. Journal of New Music Research 32(4): 429–440. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.20.3098&rep=rep1&type=pdf (accessed on 14 December 2012).Google Scholar
Jensenius, A.R, Godoy, R, Wanderley, M.M 2005. Developing Tools for Studying Musical Gestures within the Max/MSP/Jitter Environment. Proceedings of the 2005 International Computer Music Conference. Barcelona/San Francisco: ICMA, 282–5.Google Scholar
Jordá, S. 2005. Digital Luthery: Crafting Musical Computers for New Musics, Performance, and Improvisation. PhD dissertation, Universidad Pompeu Fabra (Barcelona).Google Scholar
Kenner, C. 2010. GlovePIE. http://www.glovepie.org (accessed on 25 February 2013).Google Scholar
Kiefer, C., Collins, N., Fitzpatrick, G. 2009. Phalanger: Controlling Music Software With Hand Movement Using A Computer Vision and Machine Learning Approach. Proceedings of the International Conference on New Interfaces for MusicalExpression, 2009. Pittsburgh: 246–9. http://sro.sussex.ac.uk/2202/1/ChrisKieferNIME2009Paper.pdf (accessed on 8 March 13).Google Scholar
Krefeld, V. 1990. The Hand in the Web: An Interview with Michel Waisvisz. Computer Music Journal 14(2): 28–33.Google Scholar
Levitin, D.J., McAdams, S., Adams, R.L. 2002. Control Parameters for Musical Instruments: A Foundation for New Mappings of Gesture to Sound. Organised Sound 7(2): 171–189.Google Scholar
McMullan, J. 2003. libp5glove. P5 Glove library for low level access of raw data. http://noisybox.net/computers/p5glove/libp5glove_svn_20050206.tar.gz(accessed on 12 December 2012).Google Scholar
McMullan, J. 2008. P5glove. http://noisybox.net/computers/p5glove/ (accessed on 24 December 2012).Google Scholar
Miranda, E.R., Wanderley, M.M. 2006. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Middleton, WI: A-R Editions.Google Scholar
Mitchell, T., Heap, I. 2011, SoundGrasp: A Gestural Interface for the Performance of Live Music. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo. http://www.nime2011.org/proceedings/papers/M13-Mitchell.pdf (accessed on January 2013).Google Scholar
Mitchell, T., Madgwick, S., Heap, I. 2012. Musical Interaction with Hand Posture and Orientation: A Toolbox of Gestural Control Mechanisms. Proceedings of the International Conference on New Interfaces for Musical Expression, 2012. Ann Arbor. http://www.eecs.umich.edu/nime2012/Proceedings/papers/272_Final_Manuscript.pdf (accessed on 11 January 2013).Google Scholar
Momeni, A., Henry, C. 2006. Dynamic Independent Mapping Layers for Concurrent Control of Audio and Video Synthesis. Computer Music Journal 30(1): 49–66.Google Scholar
Momeni, A., Wessel, D. 2003. Characterizing and Controlling Musical Material Intuitively with Geometric Models. Proceedings of the International Conference on New Interfaces for Musical Expression, 2003. Montreal. http://www.nime.org/proceedings/2003/nime2003_054.pdf (accessed on 14 December 2012).Google Scholar
Mulder, A. 1994. Virtual Musical Instruments: Accessing the Sound Synthesis Universe as a Performer. Proceedings of the First Brazilian Symposium on Computer Music: Caxambu, Minas Gerais: 243–50. Online: http://www.xspasm.com/x/sfu/vmi/BSCM1.pdf (accessed on 24 December 2012).Google Scholar
Mulder, A.G.E. 1996. Getting a GRIP on Alternate Controllers: Addressing the Variability of Gestural Expression in Musical Instrument Design. Leonardo Music Journal 6: 33–40.Google Scholar
Mulder, A., Fels, S., Mase, K. 1997. Empty-Handed Gesture Analysis in Max/FTS. Proceedings of Kansei: The Technology of Emotion, AIMI International Workshop. Genova. http://hct.ece.ubc.ca/publications/pdf/mulder-fels-mase–1997.pdf (accessed on 14 December 2012).Google Scholar
Mulder, A., Fels, S., Mase, K. 1999. Design of Virtual 3d Instruments for Musical Interaction. Graphics Interface ‘99: 76–83.Google Scholar
Oliver, J. 2010. The MANO Controller: A Video Based Hand Tracking System. http://www.jaimeoliver.pe/pdf/MANO-joliverl.pdf (accessed on 20 December 2012).Google Scholar
Ortiz, M., Coghlan, N., Jaimovitch, J., Knapp, B. 2011. Biosignal-Driven Art: Beyond Biofeedback. Sonic Ideas – Ideas Sónicas 3(2): 43–52.Google Scholar
Ostrowski, M. 2010. Congeries 2. https://soundcloud.com/#harvestworks/matthew-ostrowski-congeries-2 (accessed on 25 February 2013).Google Scholar
Paine, G. 2009. Towards Unified Design Guidelines for New Interfaces for Musical Expression. Organised Sound 14(2): 142–155.Google Scholar
Roads, C. 1996. The Computer Music Tutorial. Cambridge, MA: The MIT Press.Google Scholar
Rovan, J.B. 2009. COLLIDE: For Custom Glove Controller, Interactive Computer Music & Interactive Video (2002). http://vimeo.com/2729606 (accessed on 27 February 2013).Google Scholar
Rovan, J.B. 2010. Butch Rovan. http://www.soundidea.org/rovan/research_glove.html (accessed on 27 February 2013).Google Scholar
Rovan, J.B., Wanderley, M.M., Dubnov, S., Depalle, P. 1997. Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance. Proceedings of Kansei: The Technology of Emotion, AIMI International Workshop. Genova. http://www.ircam.fr/equipes/analyse-synthese/wanderle/Gestes/Externe/kansei_final.pdf. (accessed on 20 December 2012).Google Scholar
Sapir, S. 2000. Interactive Digital Audio Environments: Gesture as a Musical Parameter. Proceedings COST-G6 Conference on Digital Audio Effects (DAFx'00): 25–30. http://profs.sci.univr.it/~dafx/Final-Papers/pdf/Sapir.pdf (accessed on 13 December 2012).Google Scholar
Singer, E. 2010. Cyclops. http://cycling74.com/products/cyclops/ (accessed on 18 December 12).Google Scholar
Skogstad, S, de Quay, Y, Jensenius, AR 2011. OSC Implementation and Evaluation of the Xsens MVN suit. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 300–3.http://www.nime2011.org/proceedings/papers/G23-Skogstad.pdf (accessed on 24 December 2012). See also: http://www.youtube.com/watch?v=KI6BF_RRxQs (accessed on 19 December 2012).Google Scholar
Smalley, D. 1986. Spectro-Morphology and Structuring Processes. In S. Emmerson (ed.) The Language of Electroacoustic Music. Basingstoke: Macmillan, 61–93.Google Scholar
Smalley, D. 1997. Spectromorphology: Explaining Sound-Shapes. Organised Sound 2(2): 107–120.CrossRefGoogle Scholar
Sonami, L. 2010. Why __ dreams like a loose engine (autoportrait) (2000–8). http://vimeo.com/11316136 (accessed on 27 February 2013).Google Scholar
Sonami, L. 2013. Home Page. http://www.sonami.net (accessed on 27 February 2013).Google Scholar
Spasov, M. 2009–10. ENACTIV. http://www.keele.ac.uk/music/people/miroslavspasov/ (accessed on 17 December 12).Google Scholar
Spasov, M. 2011. Music Composition as an Act of Cognition: ENACTIV – interactive multi-modal composing system. Organised Sound 16(1): 69–86.Google Scholar
Steiner, M. 2006. Data Glove Concert. http://www.youtube.com/watch?v=uleNZXNACJM (accessed on 25 February 2013).Google Scholar
Torre, G. 2012a. Molitva (2009). http://vimeo.com/44650248 (accessed on 27 February 2013).Google Scholar
Torre, G. 2012b. Mani (2010). http://vimeo.com/44762821 (accessed on 27 February 2013).Google Scholar
Torre, G. 2012c. Agorá (2012). http://vimeo.com/44650248 (accessed on 27 February 2013).Google Scholar
Torre, G. 2013. The Design of a New Musical Glove: A Live Performance Approach. Unpublished PhD thesis. University of Limerick.Google Scholar
Torre, G., Torres, J., Fernstrom, M. 2008.The Development of Motion Tracking Algorithms for Low Cost Inertial Measurement Units. Proceedings of the Conference on New Interfaces for Musical Expression, 2008. Genova. http://nime2008.casapaganini.org/documents/Proceedings/Papers/192.pdf (accessed on 14 January 2013).Google Scholar
Vertegaal, R., Ungvary, T., Kieslinger, M. 1996. Towards a Musician's Cockpit: Transducer, Feedback and Musical Function. Proceedings of the 1996 International Computer Music Conference. Honk Kong/San Francisco: ICMA, 308–11.Google Scholar
Vigliensoni, G., Wanderley, M.M. 2009. Soundcatcher: Explorations in Audio-Looping and Time-Freezing Using an Open-Air Gestural Controller http://www.music.mcgill.ca/~gabriel/courses/mumt620/Final_Project/SoundCatcher_Final_Report.pdf (accessed on 2 June 2013). See also http://vigliensoni.com/blog/soundcatcher (accessed on 6 February 2013).Google Scholar
Vigliensoni Martin, G. 2010. The Enlightened Hands: Navigating through a Bi-Dimensional Feature Space Using Wide and Open-Air Hand Gestures. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Sydney. http://www.music.mcgill.ca/~gabriel/courses/mumt619/The_Enlightened_Hands/The_Enlightened_Hands_FINAL_REPORT.pdf (accessed on 6 February 2013). See also http://vimeo.com/10721519 (accessed on 6 February 2013).Google Scholar
Waisvisz, M. 2003. Michael Waisvisz: The Hands. http://www.youtube.com/watch?v=SIfumZa2TKY&feature=related (accessed on 27 February 2013).Google Scholar
Wanderley, M.M. 2001. Gestural Control of Music. Proceedings of the International Workshop Human Supervision and Control in Engineering and Music. Kassel. http://www.engineeringandmusic.de/individu/Wanderley/Wanderley.pdf (accessed on 13 December 2012).Google Scholar
Waters, S. 2000. The Musical Process in the Age of Digital Intervention. Ariada 1. http://www.ariada.uea.ac.uk/ariadatexts/ariada1/content/Musical_Process.pdf (accessed on 24 December 2012).Google Scholar
Wessel, D., Wright, D., Schott, D. 2002. Intimate Musical Control of Computers with a Variety of Controllers and Gesture Mapping Metaphors. Proceedings of the International Conference on New Interfaces for Musical Expression, 2002. Dublin. http://opensoundcontrol.org/files/NIME02WesselWrightSchottDmo.pdf (accessed on 14 December 2012).Google Scholar

Fischman Supplementary Material

Movie 1

Download Fischman Supplementary Material(Video)
Video 3 MB

Fischman Supplementary Material

Movie 2

Download Fischman Supplementary Material(Video)
Video 4.2 MB

Fischman Supplementary Material

Movie 3

Download Fischman Supplementary Material(Video)
Video 3.2 MB

Fischman Supplementary Material

Movie 4

Download Fischman Supplementary Material(Video)
Video 75 MB