2. RELATED WORK: COMPROVISATIONAL APPROACHES

Comprovisation is an emerging artistic strategy that encompasses composition and musical improvisation. The term comprovisation does not have a consensual definition yet. Its origins go back to the improvisational proposals of the 1960s – including the output of artistic groups such as the Scratch Orchestra and Musica Elettronica Viva (Cardew Reference Cardew1969; Curran and Teitelbaum Reference Curran and Teitelbaum1989). Current comprovisational approaches have also addressed aspects of interactive musical representations (Hannan Reference Hannan2006; Dudas Reference Dudas2010; Fujak Reference Fujak2011). Fujak (Reference Fujak2011) argues that in the activities done by living beings, some scenarios can only be partially planned. A variety of uncontrolled factors can impact the environmental conditions, shaping the behaviours of the stakeholders. Thus, he proposes comprovisation as a metaphor for ways of living. In ecological parlance, interaction units are defined as events which occur in the context of habitats and ecologies (Keller Reference Keller2000; Keller and Capasso Reference Keller and Capasso2006). When resources are deployed, the stakeholders engage in both planned actions and unpredictable behaviours. Plans are ways of projecting relationships between the extant resources, encompassing both material and cognitive activities, which may entail either teleological or exploratory strategies. In this article, we aim to construct a conceptual framework bridging comprovisation and ecologically grounded creative practicality within the ubimus context. To achieve this, we draw parallels with a conceptual tool introduced by Aliel, Keller and Ferraz (Reference Aliel, Keller and Ferraz2018b) and Messina and Aliel (Reference Messina and Aliel2019, Reference Messina and Aliel2023), the Gelassenheit.

Gelassenheit is a term coined by Heidegger (Reference Heidegger1966). Its literal translation is ‘serenity’ although Heidegger’s formulation transcends the literal meaning of this word in English. He proposes two forms of thought: 1) calculative thinking, understood as a science-oriented artistic method involving measuring, collecting data and replicating results; and 2) its counterpart, meditative thinking, entailing an attitude open to unpredictable actions and unexpected outcomes, that is, ‘an openness to mystery’ (Heidegger Reference Heidegger1966: 55). Heidegger proposes Gelassenheit as a stage to be achieved through innovative ways of thinking. From an ecologically oriented perspective and grounded on arguments by Donald (Reference Donald and Turner2006), Aliel, Keller and Costa (Reference Aliel, Keller and Costa2018a) reframe this proposal as a process of adaptation and modification of self-reflective strategies in artistic practice. The absence of control encouraged by Gelassenheit factors tends to trigger unexpected results, setting an atypical frame for potential actions and forcing the stakeholders to adapt their behaviours to unfamiliar conditions. Thus, Gelassenheit may be viewed as a factor fostering evolutionary processes within highly compressed time frames.

Within the concept of Gelassenheit defined by Heidegger, encompassing calculative thinking and meditative thinking, we can hypothesise a method for comprovisation that engages in a dialogue with the ecological perspective through action plans (guideline plan and contingency plan) (Aliel Reference Aliel2022). From an ecological perspective, therefore, comprovisation may be defined as a way to handle creative actions that feature guideline plans (that constitute the core of compositional activities – calculative thinking) and contingency plans (involving elements of improvisation – meditative thinking), having a direct relation to the local environment as source material and involving experimentation within its sphere of practice. A guideline plan establishes rules of interaction that are persistent and depend on material factors such as mass, form and the dynamic properties of materials, acting upon specific structures and resources (such as place and stakeholders). A contingency plan may deal with the same targets as the guideline plans, though it also encompasses volatile resources and processes that lack predictability. A guideline plan includes rules, planned actions, algorithms and resources often obtained through asynchronous methods (and usually linked to pre-compositional activities). A contingency plan targets the unpredictable, the eventualities, the chaotic (involving deterministic but highly complex outcomes) and random events (encompassing either environmental factors or computational strategies), thus highlighting the creative role of both human and non-human errors.

Depending on emerging musical and social needs, comprovisational strategies can adapt. Given specific conditions, guideline and contingency plans may offer opportunities to level the collective access to the creative processes, avoiding preconceived notions of priority of one strategy over another, as suggested by the critical discussions on the application of hierarchical and centralised compositional methods by Bhagwati (Reference Bhagwati, Lasker, Luz and Dack2008), Keller (Reference Keller2000) and Lewis (Reference Lewis2000). This flexibility tends to add weight to contingencies. Consequently, one of the challenges of ubimus design is to provide comprovisational guidelines that are effective both for persistent and volatile resources (Keller Reference Keller2014), while encouraging the exploration of the creative potential of contingencies.

Some comprovisational plans may involve elements of intentionality of the stakeholders. A guideline plan may feature an explicit representation of expected outcomes. Contingency plans usually entail a lack of completeness and rest on implicit knowledge shared through cultural traces (non-idiomatic improvisation is a case in point). Sometimes, both plans may converge. For instance, the absence of rules implies a new rule that limits sharing explicit knowledge. This suggests the usage of tacit or informal knowledge as a way to establish criteria and order to the actions of the agents. At the other extreme, disruptive tendencies in comprovisation may involve an entity imposing a contingent plan opposed to the shared concepts obtained through previous interactions.

For the purpose of creating a new specific neologism related to comprovisation work aligned with ecological practices (guideline plans and contingency plans), we propose the inclusion of the letter ‘e’ into the context of comprovisation, resulting in ‘ecomprovisation’. In this context, the ‘e’ combined with ‘co’ (eco) associates with ‘comp’ (composition) and ‘provisation’ (improvisation), thereby amalgamating three words into a fusion that aims to encompass the concept.

3. M335 PROJECT

In this section, we tackle the analysis of an artistic work, M335. Our focus is on the possibilities of using action plans (guidelines and contingencies) for the creation of an ecomprovisation. We approach the discussion of M335 from the perspective of a professional double bass player. Our analysis highlights technological advancements – involving the Handy S prototype – and approaches centred on musical aspects, such as the utilisation of extended techniques and graphical notations.

M335 is a piece for acoustic or electric bass, inspired by NASA’s discovery of an unidentified object. The object was photographed while distancing from the black hole M335. It has not been catalogued and studies have not yet reached consensus on its composition or nature. ^{Footnote 7} Einstein’s general theory of relativity implies that nothing, not even light, can escape a black hole. The event that took place in M335 pointed at a new phenomenon that remains unexplained. We employ this premise as an artistic inspiration. Metaphorically, the double bass tries to ‘escape’ the sound mass generated by the machine. However, the sonic materials are mostly controlled by the actions exerted on the instrument. This struggle between acoustic and electronic sources fosters a process of sonic immersion involving cycles of feedback between the gestures and the sounds.

3.1. Tools and materials

The computational prototype features two objects from the Pure Data GEM ^{Footnote 8} library, namely [pix_video] and [pix_data]. The [pix_video] object enables the use of a webcam connected to a computer. The [pix_data] object acquires the colour of a specified pixel in an image. This capture is defined by the RGB standard (red, green and blue). During the initial tests of a webcam attached to the double bass, we observed that colour-based capture does not necessarily yield a result directly connected to the musician’s gestures. The capture process and subsequently the sound production were imperfect, even in single-colour conditions. In other words, it was not possible to repeat or refine the capture. After several tests, we concluded that colour-based capture did not contribute to the generation of musically replicable material.

However, during the testing process we also noted that the musician’s movements of the double bass produced sounds he considered more ‘musical’ despite their random qualities. Based on this premise, we proposed an indirect-capture process, allowing more freedom to move along x, y and z axes. Rather than emphasising mappings for each colour; the new approach focused on the relationship between light and darkness enabled by the movements. The musician moved the double bass, the webcam captured the light sources and sounds were produced. Darkness or low illumination yielded no capture and no sound. This approach provided a baseline control of sound ‘on and off’.

Although we had a certain level of control, we considered incorporating randomness under unexpected conditions. In this context, the musician interacts with a sound (on or off) but has no control over other sonic characteristics, forcing them to adapt to the local conditions. Considering the object’s ability to capture RGB, we altered the equation of the three colours (multiplications in some objects) to generate variations in duration, velocity and pitch. ^{Footnote 9}

3.1.1. Graphics: target clear and accessible indications for interpretations and actions by the musicians/performers

We use spectrographic scores ^{Footnote 10} to support the process of aesthetic decision-making (Figure 1). Our knowledge-sharing strategy features conceptual hints rather than rigid procedures. We use two graphs to trigger alternative sonic interpretations of data captured by the satellites LIGO Livingston and LIGO Hanford. ^{Footnote 11} These sonic sources are colloquially termed ‘black-hole sounds’.

Figure 1. Spectrographic scores used in M335.

To establish a connection between the spectrogram and instrumental actions, we subdivide each spectrum in three regions. These subdivisions (indicated by green, red and blue lines) are associated with the regions of the double bass’s body and indicate where the left hand should be placed to handle pitch-oriented actions. To facilitate the comprehension by the performer, we have related the increase in intensity (displayed in the spectra) to dynamic references applicable to each segment. The double bassist can choose to play either the left side (LIGO Livingston) or the right side (LIGO Hanford). She can also switch sides at any time between LIGO Livingston or LIGO Hanford, or she may repeat the material indefinitely.

The spectrographic scores are read from top to bottom. Timbral variations are directly associated with bow movements. Thus, if a parameter value is large, the bow gesture will also be large. The wavy notation represents the intensity of the right hand (applied while bowing, pizzicato and so on). This parameter can be read from top to bottom or from bottom to top. Changes in the order during the performance are left to the discretion of the instrumentalist. There is no time restriction for repetitions, but there are restrictions on the dynamics and the strings choices. For instance, on the left side (LIGO Livingston), the strings E and A are indicated, while on the right side (LIGO Hanford), the strings D and G are indicated. As already mentioned, the choice of sides is free.

From a total of eight, four pages are used for each performance. Some pages permit a free choice of pitches, while others have guidelines indicating the pitches to be played or excluded. There are no time indications for page turns. Hence, the instrumentalist may choose to explore one or more pages before handling the complete set.

3.1.2. Extended techniques: furnish accessible and clear indications for interpretations and actions by the performers

Extended techniques based on explicit instructions are featured as part of the guidelines plan. Despite adopting a limited set of instrumental techniques, we tailored the design for: 1) adaptation of sonic processing based on bow movements; and 2) adaptations tailored for the sonification model (more on this later). The variability is governed by rules of action and reaction that depend on the local context. Consequently, small-scale decisions play a decisive role. The implication is that a creative agent cannot expect to replicate previous results because there is no univocal relationship between the audio-processing techniques and the parameters driven by this form of performance. ^{Footnote 12}

In the performance, we can observe the musician utilizing some of the proposed features, placing emphasis on the body movement gesture of the double bass to activate the webcam feature, employing extended techniques, and at times referring to the score to guide certain actions (Figure 2; Video Example 1).

Figure 2. M335’s performance.

3.1.3. Usage of the Handy S prototype in M335

The prototype Handy S was used to explore adaptive strategies (Keller, Aliel and Silva Reference Keller, Aliel and Silva2018): 1) parametrisation based on the capture of acoustic-instrumental gestures – a webcam is coupled to the instrument providing a gestural sonic interface that bypasses the typical process of acoustic-instrumental transduction; and 2) the generative sonic results are recycled as material for improvisation.

We conducted preliminary studies that led to the deployed version of M335, involving performers accustomed to improvisation practices who were familiar with contemporary musical repertoire. These studies pointed to strategies to attain greater control of the sonic outcomes: it was observed that some performers began to track the timing of the webcam ‘errors’. Thus, they learned to anticipate when the system generates an update and employed these opportunities to try new behaviours leading to new sounds.

Another strategy involved learning that small changes of position in the x, y and z axes result in greater control of the outcomes. Instead of using manual gestures to trigger modifications (following their standard instrumental training), the performers tried moving the body of the instrument. Since the webcam is attached to the instrument, the movements of the double bass generated more precise sonic outcomes. We can interpret these strategies as human behaviours adapted to specific demands of the ubimus ecosystem. We believe these features were discovered because the musicians searched for apparent flaws in the system that allowed certain materials to be ‘tamed’, becoming more predictable and controllable.

4. M335: SUMMARY OF FINDINGS

M335 features two forms of auditory-display: action and feedback. The interaction model involves reading and interpreting a score and capturing visual data to transition from no sound to sound. The complete cycle involves events initiated by the instrumentalist, visual data captured by the webcam, and sonic outcomes that are not fully determined by the state of the system. While executing the score, the performer triggers multimodal events (Figure 3). ^{Footnote 13} The spectrographic scores are used as a guide for the improvisational processes (see Opie and Brown Reference Opie and Brown2006; Connors Reference Connors2015 for related techniques). These processes do not imply a direct mapping between data and sonic outcome: they yield a new sonic layer that diverges from the acoustic-instrumental material. Furthermore, the player cannot intentionally replicate the processed material because the behaviours are shaped though sonic feedback. Instead of using only the sound of the instrument or applying extended techniques, our ecomprovisational approach promotes the empirical exploration of electroacoustic and acoustic sonic resources.

Figure 3. Diagram of the M335 interaction model.

A second strategy aims to increase the creative relevance of the sonic sources. A fairly entropic level is fostered through a ‘reversal’ of roles. The musician (although still triggering audio processes) relies on events produced by the interactions with the Handy S prototype that may not necessarily involve instrumental sound. The instrumental gestures are captured by the webcam (Figure 4), ^{Footnote 14} triggering processes that may diverge from what is expected by the instrumentalist. Consequently, electronic sounds become central. The outcomes are linked to the interactions between the instrumental sound and the movement-tracking system. Through this dual and synchronous interaction, the audioprocessing parameters introduce new and divergent material based on the double bass sounds.

Figure 4. A consumer-level camera was attached to the double bass to enable data capture and parametric control in M335.

As previously explained, the current prototype retains a certain level of randomness due to the low definition of the visual data and due to the automated ‘choices’ of data capture. These factors, rather than being treated as limitations, constitute a target of design for the M335 ecomprovisational project. Although not planned as a design feature, during the sessions the musician pointed out that the projection of the hands of the performer influenced the way they were playing. At a later stage, this third strategy was integrated into the design (see Aliel Reference Aliel2017). Through the projection of the movements of the hands, the unintentional material is reframed, grounding the assessments of the contingencies on multimodal information and enhancing the immersion in the ecomprovisational experience.

From this perspective, we consider that the M335 prototype follows the proposal by Boden and Edmonds (Reference Boden and Edmonds2009) to apply constraints to promote complex and creative results. These constraints serve to frame the performance leaving room for ‘computational choices.’ ^{Footnote 15} Furthermore, a connection arises between Heidegger’s calculative and meditative thinking, as presented earlier. While calculative thinking permeates the aesthetics and practice based on the acoustic instrument, expanded through the option to turn the electronic sounds on or off, meditative thinking is encouraged by fostering openness to the unexpected and adapting to the local conditions.

If we consider the application of Gelassenheit, we can suggest that the occurrence of unexpected conditions (almost with random characteristics) is not actually a problem. In general, this kind of event is common in ecological conditions (natural events that cannot be controlled by humans or computers). Thus, from this perspective, unexpected elements can bring unique benefits to the process and can be understood as events within the contingency plan. In other words, even a technological parametrisation error will be interpreted as a form of ‘improvisation’. We term these conditions Gelassenheit events (Aliel Reference Aliel2022).

5. FINAL REMARKS AND FUTURE DEVELOPMENTS IN ECOMPROVISATION

Does ecomprovisation provide a path to deal with the demands of future ubimus initiatives? Let us take a moment to reflect on the proposals laid out in Keller, Messina and Oliveira, Reference Keller, Messina and Oliveira2020a and Keller, Simurra and Messina, Reference Keller, Simurra and Messina2020b targeting second-wave ubimus endeavours. Second-wave ubimus may encompass everyday creative musical activities and collective asynchronous musical interaction. The constraints of casual interaction encouraged by ubimus initiatives include: 1) very limited time for preparations; 2) short-term engagements; 3) a wide diversity of stakeholder profiles; 4) unreliable network connectivity; and 5) unrestricted public exposure. While some of these caveats are addressed by the techniques discussed in this article, others constitute targets of future research. Despite envisaging professional-creativity support, some of the strategies implemented in M335 could be adapted to address the needs of casual participation (Keller et al. Reference Keller, Aliel and Silva2018). In particular, the techniques developed for the creation of the artwork M335 provide ways to extract knowledge from local resources that can enhance improvisational approaches. These approaches may potentially include strategies for everyday musical creativity for both trained and untrained participants in the future.

Potential future approaches include:

1. 1. By using open graphic scores that present basic instructions for actions, the gap between learning and performing is reduced, allowing both groups to employ similar resources (see Aliel Reference Aliel2022).

2. 2. Since the support involves coupling a technological object to an acoustic instrument, the process of gestural and sonic experimentation could eventually unfold similarly for both novices and more experienced participants, fostering knowledge exchanges through mimicking or imitation.

3. 3. As unpredictable factors are an important feature of ecomprovisation, both groups need to deal with emergent properties by finding solutions or incorporating new actions into their palette of creative tools. This implies a pressure on the boundaries between everyday musical creativity and professionally oriented approaches. Ecomprovisation may furnish a bridge to explore both perspectives.

4. 4. In line with the maker-oriented DIY ubimus proposals (Lazzarini, Keller, Otero and Turchet Reference Lazzarini, Keller, Otero and Turchet2020), everyday objects could be repurposed through ecomprovisational strategies. ^{Footnote 16} These techniques could bypass several of the financial barriers faced by potential participants in low-income regions. Owing to the scarcity of professional audiovisual equipment in peripheral countries, gear such as high-resolution video-cameras may be replaced by recycled consumer-level mobile telephones. Android systems, for example, offer a wide range of free or open-source software ^{Footnote 17} that support synchronous streaming of audiovisual data to personal computers.

5. 5. The incorporation of more refined movement-tracking algorithms (as exemplified by Chakraborty, Yaseen, Timoney, Lazzarini and Keller Reference Chakraborty, Yaseen, Timoney, Lazzarini and Keller2022) also opens interesting possibilities to release the interaction from the bounds of the acoustic instruments. Mid-air interaction techniques involve triggering a multiplicity of sources from a visual-tracking algorithm that supports adaptations to diverse body shapes and positions. No special-purpose equipment is necessary.

Finally, a note of caution is necessary regarding the support for distributed activities. Collective asynchronous musical activities bypass co-located face-to-face interactions. Consequently, they may reduce the social-bonding aspects fostered by co-located music-making. While several comprovisational proposals have targeted synchronous activities, open temporalities tend to be linked to non-hierarchical and asynchronous techniques. Ubimus ecomprovisational techniques could provide a way to deal with asynchronicity and quasi-synchronicity in comprovisation. On the one hand, group music-making lets the stakeholders build a shared knowledge pool that grounds consensual decisions. Alternatively, it fosters the coexistence of divergent worldviews. On the other hand, the participants’ usage of the local resources encourages aesthetic diversity. Diversity within consensus and coexistence of opposed views seem to be attitudes encouraged by ubimus endeavours. Ecomprovisational strategies – as exemplified in M335 – feature forms of interaction aligned with these goals.