Skip to content

Categories:

EIM-Live provocation

EIM-Live (embodied intelligence in music): a provocation

In general, Artificial Intelligence (AI) and music research has mainly focused on machines with the ability to ‘behave liked skilled musicians’, or that ‘exhibit behaviours that unbiased observers would deem to be creative’ (Miranda 2000; Guerra 2002; Colton & Wiggins 2014). Additionally, it has favoured machines that perform some analytics and synthesis, such as the autonomous composition in the style of x (Pachet, 2002; Rolland and Ganascia 2000), or some other operation in which the ‘machine equals the brain’ (Jefferson 1949). This understanding of intelligence foregrounds thought and reason, and adopts approaches based on the creation of software or machine tools that focus on representation and the application of high-level cognitive skills such as problem solving, planning, or aesthetics (Anderson 2003, Miranda 2000, Colton 2009, Moffat & Kelly 2006).

While such ‘mind’-based research serves a valuable purpose of building a virtual performer that ‘thinks’ like a musician, it fails to capture one of the most important aspects of musicianship: embodiment. Only through an embodied experience may a musician ‘feel’, as well as ‘think’, the music. EIM-Live research therefore places an emphasis on the relationships between human and machine musicians, focusing on their creative partnerships in co-operative music-making. In April 2016 a network of trans-disciplinary researchers from computational intelligence, creative computing, embodied cognition in music, software philosophy, media cognition, computer game theory, and music composers, improvisers and performers gathered to discuss this emerging field. The result of this meeting was a collective provocation towards the next steps of EIM-Live research. This provocation makes four points:

  1.     That embodied intelligence is a process in the “Flow” of music-making where human and machine performers work together, or take part in the act of “musicking”
  2.     That co-operative EIM-Live music-making relies on tacit and epiphenomenal responses and operates within a social context
  3.     That the temporal presence and operativity of technological media is a fundamental agent in the relationship between EIM-Live humans and machines
  4.     That EIM-Live requires new models of understanding how consciousness and being are negotiated in the co-operative relationship between human, machine, media and code.

 

  1. That embodied intelligence is a process in the “Flow” of music-making where human and machine performers work together, or take part in the act of “musicking”

Embodied music cognition research has found that an embodied interaction with music ‘implies the corporeal attunement of the musician to the sonic event that results from the performance’. As such, the whole experience of participating within a music-making environment creates an ‘optimal embodied experience (‘Flow’) when the musician is completely immersed in the created musical reality (presence) and enjoys himself through the playfulness of the performance’ (Nijs, Lasaffre, Leman 2013).

EIM-Live researchers Emmerson and Vear corroborated this from their own research perspectives. Emmerson (2015) suggests that we are misunderstanding the limits of ‘machine listening’, and argued for a partnership of human and machine, a paradigm where the machine’s function was to enhance and extend the embodied human experience. Vear (2014) discusses his practice-based research that investigates the co-operative relationship between thinking-machines and human performers, within the act of music-making. Vear argues that the machines collaborate within the ‘Flow’ of music, and that their “intelligence” is embodied within this realm of musicking (Small 1998).

For EIM-Live researchers, making music is understood not to be an activity removed from ourselves (i.e. projecting a sound outwards into the world), but it is a state of becoming within this flow. To clarify this EIM-Live research is staking a claim at the intersection of embodied phenomenology and ideas in digital game studies regarding embodiment (Klevjer 2006; 2012; Bayliss 2007) and incorporation (Calleja 2011). Within this, Daniel Vella argues that digital game play locates the player in an embodied ludic subjectivity in which the immersant is invited to explore and perform through play (Vella 2015; 2016).

  1. That co-operative EIM-Live music-making relies on tacit and epiphenomenological responses and operates within a social context

Michael Young argues that computational systems able to collaborate with human improvisers can be termed live algorithms: systems that can cooperate proactively, on an equal basis, with musicians (Young and Blackwell 2013). Live algorithms must have a demonstrable autonomy and abilities to both react and contribute constructively within the “Flow” of music. For EIM-Live research this means that contributions seem equally valuable, both as independent statements and as offerings to the collective that might invite response from other machine or human participants. EIM-Live acknowledges that this definition is an ideal, and it raises fundamental questions about creativity and group interaction and how these might be computationally modelled. The aim is to develop live algorithms that generate output independently of performer and designer, but with a capacity to be influenced by the collaborative social environment of music-making they inhabit.  The computational methods to render the appearance of independence might lie outside of any model of human cognition.

        The composer/ musician Sebastian Lexer questions what can learning machines offer to human performance. He argues that human performance, particularly in its improvised form, evolves around the lucidity of its underlying rule sets and their subsequent subversion, rejection or negation in practice. Performing humans share a substantial body of musical knowledge (of styles, harmonies and structures), which enables them to work together within the “Flow” without prior discussion. For EIM-Live research, the detection of patterns in behaviour is key to the human psychological drive towards change (Kuhl, 2001) and stimulates the human capability to think about potentiality (Agamben 1999). In terms of musical performance human thought goes beyond what has been performed and is intrinsically linked to the performer’s ‘inner world’ (Sloterdijk, 2004). It is from within this perspective that EIM-Live presents the question: How would the machine’s potentiality be communicated and perceivable?

Thor Magnusson goes further by questioning the change in the social and cultural context in which music of today has meaning, perhaps diverging from the paradigms of our recent history through gigs, CD’s, tracks, downloads and music videos. He posits that home-brew and boutique music-making is the emergent culture of today and that EIM-Live should look at the society created in these realms. Hack labs and maker fairs are ideal testing grounds for development in AI and machine learning. EIM-Live recognises that music is sharing an increasingly popular platform of cultural experience, where the focus appears to be shifting to more sense intensive media where AI, VR, immersiveness, collaborating crowds, and other new forms of cultural experiences are taking the fore. For this reason, it is important for EIM-Live to clearly communicate the underlying algorithms to the audience, to open up and share the technologies that drive the musical work.

  1. That the temporal presence and operativity of technological media is a fundamental agent in the relationship between EIM-Live humans and machines

EIM-Live researcher David M. Berry advocated Wolfgang Ernst’s Chronopoetics (time-giving) as a central philosophical framework for understanding the technological analysis of media-time processes within EIM-Live. A study of time-critical moments within media technologies, and how media temporalities affect and effect traditional human perceptions of the sense-of-time, is central to understanding how the ‘temporal essence enriches the tradition of philosophical enquiry into the nature of time’ (Ernst 2016).

Code-based systems can be the primary interface and operational media through which we design the temporal being of digital music systems. Rebecca Fiebrink proposes the need for other design tools (e.g., machine learning) to complement code if we are to design musical systems that act as partners to humans rather than servants. Demonstration-based interfaces for human creation of digital systems (e.g. built with machine learning) can take advantage of people’s embodied understanding of temporal processes, and their ability to act out parts in those processes. For instance, their ability to demonstrate relationships between movement and sound over time, or between two musical partner agents over time. Contrast this with code, which requires people to abstract their understanding of temporal processes into a set of rules or procedures that exist outside of time.

For EIM-Live researchers, code in itself, is not as well suited for expressing embodied and tacit knowledge or for engaging holistic and exploratory design activities. The challenge we present is to create systems that surprise and challenge people in meaningful ways which may have a direct relationship with the ability to embed embodied knowledge or ideas within a system and therefore a co-operative relationship. James Saunders presents the notion of ‘software for people’ (after Oliveros), in which composers work with verbal notation that embed a sense of learning through experience. In this way, the musicians running the programme bring forth epiphenomenological interpretations of the code which is often holistic, exploratory unpredictable and challenging as a result, This, in turn, presents composers and coders with insights into the tacit knowledge of machine-intelligent systems, from the perspective of the musician.

For the composer and improvising performer Christopher Redgate temporal essences are of fundamental importance, as “I want to make journeys with the music and not simply amble along”. A central concern for Redgate is meta-structure, in which temporal frameworks have a sense of audible or discernible overarching structure, however abstract these may be. For the performer Angharad Davies, time is inextricably linked to musical and physical space. Before a performance can take place, especially improvised, a collective wish to share time and space to make music and to listen to music must coincide. Alongside this is a curiosity for exploring the acoustic and natural tuning of the performance space. Negotiating a balance between instinct and inspiration involves employment of any of the following strategies – use of a stop-watch, limiting or fixing the amount of sounds or preparations to be used, using unstable and unpredictable preparations. When improvisation goes well a feeling of total immersion in the sound occurs where the sound leads and playing is unfolding in the moment.

        James Saunders argues for a re-interpretation of the time-critical processes of machines by re-positioning them into a temporal scale comprehendible by humans. By employing human-computational approaches in performance we get to understand them for their intrinsic complexity, and can co-operate with them as temporal beings within human perceptions of the “Flow” of music. Significantly, from a creative computing perspective, it is possible to introduce humanness into the code through play, choice, aesthetic or mischief. Berry, on the other hand, argues that the super-fast processes of computation, and the multiplicity of temporal operativity across the computational architecture can, in themselves, present new models for composition and temporal organisation. EIM-Live research advocates both interpretations.

  1. That EIM-Live requires new models of understanding how consciousness and being are negotiated in the co-operative relationship between human, machine, media and code.

In recent years, the nature of consciousness has been reconsidered and re-conceptualised to accommodate the transformational relationships between technology and the post-human. The idea of the Cartesian mind-body duality is being challenged by an understanding that consciousness is embodied, and that cognition is a situated activity. The focus here for EIM-Live is not on thought and reason, but on the wholeness of experience, and suggests that thinking beings should “be considered first and foremost as acting beings”, where agency and interactive coping occupy centre-stage (Anderson 2003).

With this in mind, Vear’s hypothesis is that making music is not an activity removed from ourselves (i.e. projecting a sound outwards into the world), but it is a state of becoming. Insofar that musicians do not do music, but become the music they make. This process of becoming he perceives to be similar to a gamers incorporation into their gameworld, or the gameworlds assimilation into their world, and represents a full embodied shift into their sound. With this in mind, we should consider that when musicians make music together it is them as sound that are interacting, enmeshing, co-operating, conjoining, dancing within a dimension called music. So, the challenge to EIM-Live research is to find a way of implementing intelligent systems into this dimension of music-making, that STIMULATES this experience  (not represents or simulates).

Bennett Hogg argues that although technologies can extend capabilities and compensate for lack, attempts for a seamless over-integration is not necessarily the most productive strategy, artistically or politically. For the surrealists the automaton decentred, in its uncanniness, the bourgeois norms of rationalistic thought, in a parallel move a century and a half earlier, La Mettrie’s Man, a Machine undermined the claims of Church and Monarch to rule, grounded upon a sovereignty over the soul. Machines are other. The life-presence within their otherness can stand as a reminder – an insistence – against any tendencies to technological narcissism and the consequent anthropocentrism and dominance of the shared ecosystem.

In line with current critiques of consciousness and being (e.g. Braidotti 2013), the EIM-Live approach is to work creatively to critique anthropocentrism and dispositional notions of consciousness (Kittler 1997, 1999). EIM-Live acknowledges that in addition to medium theoretic approaches, actor-network theory presents fertile ideas with which to develop an exploration of how the computer might be said possesses its own unique mode of existence (see Latour 2013) and that humans are part of larger complex socio-technical systems that generate new possibilities for exploring philosophical and theoretical concepts and aesthetic experiences (see Latour 2005; Bishop 2012; Berry and Dieter 2015). Andrew Hugill argued that Integrated Information Theory (IIT) presents a compelling model towards developing an approach with which to understand consciousness in EIM-Live context, but would require further investigation. Certain areas of IIT posits a structure with which to understand a system’s consciousness by determining its causal properties, and therefore the intrinsic, fundamental properties of what these causal systems bring into being.

Hugill also argues for a ‘creative computing’ approach in which the creative partnership between humans and machines must focus for the centrality of humans. Whilst considering ‘Flow’ and consciousness he calls for the creative abuse of embodied technology by ‘designing for surprise’. EIM-Live research advocates Hugill’s assertion that a key part of developing an understanding of consciousness within creative computing systems, is to draw out the tacit knowledge of experience from within praxis. Interrogating the tacit dimension (Polanyi 1966) of praxis across the range of creative acts within EIM-Live will help to distinguish between propositional knowledge (facts), procedural knowledge (processes), and the tacit knowledge of ‘knowing more than we can tell’ (Polyani).  

 

NEXT STEPS

The next stage of EIM-Live research needs to address the question: How does Music-AI make musicians more creative? This should be conducted through a scientific study of the artistic practice of musicians within Music-AI, by measuring and correlating their embodiment in the music, their understanding of the Flow of the music, and the opportunities for innovation in the creation of Music-AI and technically mediated performance environments.

EIM-Live has identified 5 themes with which this scientific research should be articulated:

  1. Co-operation
  2. Time
  3. Machine Learning
  4. Societies
  5. Towards a Conscious Machine

Viewing EIM-Live through each of these lens will contribute new insights to the points made in this provocation, whilst providing an appropriate depth of cross-talk between each theme, and through each successive theme case study.

 

Authors:

 

Prof. David. M. Berry

Ms. Angharad Davies

Prof. Simon Emmerson

Dr. Rebecca Fiebrink

Dr. Bennett Hogg

Prof. Andrew Hugill

Dr. Sebastian Lexer

Dr. Thor Magnusson

Mr. Christopher Redgate

Prof. James Saunders

Prof. Craig Vear

Dr. Daniel Vella

Prof. Michael Young

(April-June 2016)

———————————————

 

Individual Provocations:

Angharad Davies

Before an improvisation can take place a collective wish to share time and space to make music and to listen to music must coincide. So, it’s a collaboration not only with the performers making that commitment but also that of an open minded audience that are willing to experience something new. Alongside this is a curiosity for exploring the acoustic and natural tuning of the performance space. Negotiating a balance between instinct and inspiration involves employment of any of the following strategies – use of a stop-watch, limiting or fixing the amount of sounds or preparations to be used, using unstable and unpredictable preparations. When improvisation goes well a feeling of total immersion in the sound occurs where the sound leads and playing is unfolding in the moment.

Simon Emmerson

For the 20th Birthday issue of Organised Sound I wrote ‘Listening With Machines: A shared approach’. Suggesting we were misunderstanding the limits of ‘machine listening’, I argued for a partnership of human and machine, a paradigm where the machine’s function was to enhance and extend the embodied human experience. I will now develop this to include performance with live electronics. Reference: ‘Listening With Machines: A shared approach’. Organised Sound, 20(01), 2015, pp. 68-75.

Rebecca Fiebrink

Code is the primary interface through which we design digital music systems. Code is well-suited for precisely instantiating well-defined ideas, and for specifying flows of control (e.g. mappings) that are mathematically simple (e.g., involving one-to-one, linear, deterministic processes). However, code is not as well suited for expressing embodied and tacit knowledge, for engaging holistic and exploratory design activities, or for creating systems that surprise and challenge people in meaningful ways. I propose the need for other design tools (e.g., machine learning) to complement code if we are to design musical systems that act as partners to humans rather than servants.

Bennett Hogg

Though technologies extend capabilities and compensate for lack, attempts for a seamless over-integration is not necessarily the most productive strategy, artistically or politically. For the surrealists the automaton decentred, in its uncanniness, the bourgeois norms of rationalistic thought, in a parallel move a century and a half earlier, La Mettrie’s Man, a Machine undermined the claims of Church and Monarch to rule, grounded upon a sovereignty over the soul. Machines are other. Their otherness can stand as a reminder – an insistence, in my view – against any tendencies to technological narcissism and the consequent anthropocentrism and dominance of the shared ecosystem.

Andrew Hugill

Beware the p-zombies! This provocation focuses on the creative partnership between humans and machines. It argues for the centrality of humans. It considers ‘flow’ and calls for the creative abuse of embodied technology. Finally it questions the envisaged approach to consciousness and suggests that Integrated Information Theory be considered.

Sebastian Lexer

What can learning machines offer to human performance? Human performance, particularly in its improvised form, evolves around the lucidity of its underlying rule sets, subversion, rejection and negation. The detection of patterns in behaviour is key to the human psychological drive towards change (Kuhl, 2001) and stimulates the human capability to think about potentiality (Agamben 1999). Human thought goes beyond what actually happens at any moment. In terms of musical performance – human thought goes beyond what has been performed and is intrinsically linked to the performer’s ‘inner world’ (Sloterdijk, 2004) – understanding of the world. How would the machine’s potentiality be communicated and perceivable?

Thor Magnusson
Music is moving to the periphery of cultural experience. Unlike a few decades ago where music was a central cultural reference for young people, a source for identity and ideology, the focus has now shifted to more sense intensive media, where AI, immersiveness, collaborating crowds, and other new forms of cultural experiences are taking over. What we called music and released on media formats such as LPs and CDs is disappearing, rendering the rock-gig with the same cultural status as the opera (a dead art form to be preserved in bespoke museums). In this provocation I will describe how this is happening, and in so doing I hope to provoke and make people angry, skeptical, and reject my arguments. I might reject them myself.

Christopher Redgate

Some time ago I had the opportunity to work with a very high level classical pianist who is also an exceptional improviser. From the moment we started playing together it was obvious that we thought in similar ways and shared a substantial body of musical knowledge (knowledge of styles, harmonies and structures) which enabled us to work together in a variety of musical styles (Brahms, C. P. E. Bach, Mozart, Bartok , etc.) without prior discussion. This challenged me to think about such shared knowledge and about some of its implications. Of particular interest to me as composer and improvising performer is the idea of meta-structure, frameworks which have a sense of audible or discernible overarching structure, possibly referencing ideas already used, processing these ideas etc. This seems important to me as both composer and improviser – I want to make journeys with the music and not simply amble along.

Craig Vear

Following Vear (2014), my practice-based research has investigated the co-operative relationship between thinking-machines and human performers, within the act of music-making. By that I mean, that the machines collaborate within the ‘flow’ of music (Nijs, Lasaffe, Leman (2013)), and that their “intelligence” is embodied within the realm of musicking (Small 1998). As such, making music is not an activity removed from ourselves (i.e. projecting a sound outwards into the world), but it is a state of becoming i.e. musicians do not do music, but become the music they make. And therefore, to understand where thinking machines co-operate with humans, I need to appreciate the embodiment processes that occurs when musicians become music. Reference:  Vear, C., (2014) Music, dimensions and play: composing for autonomous laptop musicians and improvising humans. Digital Creativity, 25 (4), pp. 343-356

Daniel Vella

Staking a claim at the intersection of the embodied phenomenology of Sartre and Merleau-Ponty and ideas in digital game studies regarding embodiment (Klevjer 2006; 2012; Bayliss 2007) and incorporation (Calleja 2011), this provocation argues that digital game play locates the player in an embodied ludic subjectivity she is invited to explore and perform through her playing of the game (Vella 2015; 2016).

Michael Young

Computational systems able to collaborate with human improvisers can be termed live algorithms: systems that can cooperate proactively, on an equal basis, with musicians. This means that contributions seem equally valuable, as independent statements and as offerings to the collective. This definition is an ideal, and it raises fundamental questions about creativity and group interaction and how these might be computationally modelled. Can musicians and computers relate to each other as equivalent partners? Could an audience recognise this relationship? (Young M. and Blackwell T., ‘Live Algorithms for Music’. The Oxford Handbook of Critical Improvisation Studies, Volume 2 2013).

 

—————————————————————————————-

References

Agamben, G., (1999) Potentialities. Stanford CA: Stanford University Press, 1999

 

Anderson, M., L., (2003). “Embodied Cognition: A field guide”. Artificial Intelligence, 139, 91-130

 

Bayliss, P., (2007) “Beings in the Game-world: Characters, Avatars, and Players.” In IE ’07: Proceedings of the 4th Australasian conference on Interactive entertainment (2007), pp. 1-6

 

Berry, D., M., and Dieter, M., (2015) Postdigital Aesthetics: Art, Computation and Design, Basingstoke: Palgrave.

 

Bishop, C., (2012) Digital Divide, Artforum, September 2012, accessed 08/07/2013, http://hybridge.files.wordpress.com/2011/02/bishop-digital-divide-artforum-sep-2012.pdf

 

Braidotti, R., (2013) The Posthuman, Cambridge: Polity Press.

 

Calleja, G., (2011). In-Game: From Immersion to Incorporation, USA, MIT Press

 

Colton, S., (2009) ‘Seven catchy phrases for computational creativity research’, in Proc. of the Dagstuhl Seminar: Computational Creativity: An Interdisciplinary Approach,

 

Colton, S., and Wiggins, G., A., (2012). “Computational creativity: The final frontier?” In Proceedings of 20th European Conference on Artificial Intelligence (ECAI), pages 21-26, Montpellier, France.

 

Csikszentmihaly, M., (1990)  Flow: The Psychology of Optimal Experience, New York: Harper Collins

 

Emmerson, S., (2015) ‘Listening With Machines: A shared approach’. Organised Sound, 20(01), 2015, pp. 68-75.

 

Ernst, W., (2016) (trans. Enns, A.) Chronopoetics; The Temporal Being and Operativity of Technological Media London; Rowman & Littlefield

 

Guerra C. (2002) “The Mechanization of Intelligence and the Human Aspects of Music” in Miranda, E ed.. (2000). Readings in Music and Artificial Intelligence.

 

Jefferson, G., (1949) ‘The mind of mechanical man’, British Medical Journal, 1(4616),

 

Kittler, F., (1999) Gramophone, Film, Typewriter, Stanford, CA: Stanford University Press.

 

Kittler, F., (1997) Literature, Media, Information Systems, London: Routledge.

 

Klevjer, R., (2012). “Enter the Avatar. The phenomenology of prosthetic telepresence in computer games.” In H. Fossheim, T. Mandt Larsen & J. R. Sageng (Eds.), The Philosophy of Computer Games (pp. 17-38). London & New York: Springer.

 

Klevjer, R., (2006). “La via della pistola. L’estetica dei first person shooter in single player.” In M. Bittanti & S. Morris (Eds.), Doom. Giocare in prima persona. Milano: Costa & Nolan.

 

Kuhl, J., (2001) Motivation und Persönlichkeit. Göttingen: Hogrefe-Verlag, 2001.

 

Latour, B., (2013) An Inquiry Into Modes of Existence: An Anthropology of the Moderns, Harvard University Press.

 

Latour, B., (2005) Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford: Oxford University Press.

 

Miranda, E., ed., (2000). Readings in Music and Artificial Intelligence. Publisher: Routledge.

 

Moffat D., & Kelly, M., (2006) D Moffat and M Kelly, ‘An investigation into people’s bias against computational creativity in music composition’, in Proceedings of the International Joint Workshop on Computational Creativity

 

Nijs, L., Lesaffre, M., & Leman, M. (2013). “The musical instrument as a natural extension of the musician.” In M Castellengo & H. Genevois (eds.), Music and its Instruments. Sampzon: Editions Delatour France.

 

Pachet, F., (2002). “The Continuator: Musical Interaction With Style”. In Proceedings of the 2002 International Computer Music Conference (ICMC).

 

Polanyi, M., (1966) The Tacit Dimension

 

Rolland, P.-Y., & Ganascia, J.-G. (2000). “Musical pattern extraction and similarity assessment.” In E. Miranda (Ed.), Readings in Music and Artificial Intelligence (pp. 115-144). New York: Harwood Academic Publishers.

 

Sloterdijk, P., (2004) Sphären III, Schäume. Frankfurt a. M.: Edition Suhrkamp, 2004.

 

Small, C., (1998). Musicking: The Meanings of Performing and Listening, Hanover: University Press of New England

 

Vear, C., (2014)  “Music, dimensions and play: composing for autonomous laptop musicians and improvising humans” in Digital Creativity, 25, 4, 343-356

 

Vear, C.,( 2014b) – in Davismoon, S., (2014) “Immersive, Interactive, Real and Imagined Sonic Environments: Encountering the Aural Muse in Imagined, Implied Spaces, in Intelligent Technologies for Interactive Entertainment: 6th International Conference, INTETAIN 2014 Proceedings pp115. Springer

 

Vella, D., (2015) The Ludic Subject and the Ludic Self: Investigating the I-in-the Gameworld, Ph.D thesis, IT University of Copenhagen, 2015

 

Vella, D.,  (2016)  “The Ludic Muse: The Form of Games as Art” in CounterText, Volume 2 Issue 1, Page 66-84, ISSN 2056-4406

 

Young M. and Blackwell T., ‘Live Algorithms for Music’. The Oxford Handbook of Critical Improvisation Studies, Volume 2 2013).