Datasounds: When Music Meets AI

Performers, composers, and improvisers are contributing through embodied knowledge and practice-based methods; researchers in psychology bring insights about cognitive, affective and behavioural processes underpinning musical experience; and data scientists add analytical expertise as well as relevant theories, methods and techniques.

The network is led by Dr Oded Ben-Tal (Principal Investigator, Kingston University), and Dr Federico Reuben (Co-investigator, University of York), and features network partners Professor Emily Howard (RNCM PRiSM), Professor Robin Laney (Open University), Professor Nicola Dibben (University of Sheffield), Professor Elaine Chew (King’s College London), Professor Bob Sturm (Royal Institute of Technology, KTH).

Datasounds: When Music Meets AI

A Documentary Film by the Datasounds, Datasets, and Datasense AHRC Research Network

By Oded Ben-Tal, Federico Reuben, Emily Howard, Robin Laney, Nicola Dibben, Elaine Chew, Bob Sturm

Google DeepMind’s recent release of its music generation model Lyria is the latest in a stream of headline grabbing innovations in AI music creation. However, amidst the noise, a quieter yet profound revolution has been shaping the landscape of music for a long time—the datafication of music.

Through this AHRC-funded network led by Dr Oded Ben-Tal (Kingston University) and Dr Federico Reuben (University of York), we investigated how data-rich technologies and practices can impact musical creativity and understanding:

Datasounds, datasets and datasense: Unboxing the hidden layers between musical data, knowledge and creativity

In the final stage of our project, we worked with film-maker Andrei Sas to produce a short documentary video. In the documentary, members of the Network reflect on the challenges we encountered during the course of our two-year collaboration, and share some of the insights we gained through our activities.

Reflections from the Network Partners

Dr Oded Ben-Tal (Network Principal Investigator)

Composer and Associate Professor, Kingston University

“Leading this research network was a very rewarding experience for me. First and foremost because I got to meet new people and learn about their research. When organising activities – online seminars or physical events – it might be tempting to rely primarily on our existing formal and informal networks. But I also used this opportunity to seek out new links: people I was aware of but who’s research I was not familiar with, as well as going online to actively seek new connections. As we can see in the documentary, this network already gave rise to new research collaborations putting AI technology on stage performing in dialogue with human improvisers. From January 2024 I will be extending this work with funding from the Volkswagen Foundation. Our aim is to study creative processes as they happen in real-time jointly between human and computer. This project was enabled by the new connections established through the research network.

Reflecting back on the process – from getting together with Emily, Robin and Federico to write the proposal to the current stage of writing a paper together – it is interesting how our ideas changed and perhaps how we changed. From the beginning we had the idea that digital data is important, perhaps now we are starting to understand how much it matters and have a better sense how.”

Dr Federico Reuben (Network Co-Investigator)

Composer, sound artist and live electronics performer

Associate Professor, University of York

“Through the ‘Datasounds, datasets, datasense’ network, I had the opportunity to meet world-leading researchers deeply engaging with music data in various ways and pursuing different objectives. The exchange of knowledge with experts from diverse areas and the newfound insights into using musical data across fields were enlightening and inspiring. It transformed my understanding of the relationship between musical data and the sense-making process in music, influencing and shaping my approach to working with data in creative music making. Lastly, I encountered new collaborators with whom I am presently engaged in research projects that originated from discussions during the network events.”

Professor Emily Howard (Network Partner)

Composer, Professor of Composition and Director PRiSM

Royal Northern College of Music

“I am grateful to Oded Ben-Tal and Federico Reuben for their time and work in establishing the Datasounds AHRC network, which has already had so many measurable benefits, not least in bringing a set of diverse artists and scientists together through focused conversations. I feel fortunate to have had the opportunity to make these many connections and look forward to growing them in the coming years. From a personal perspective, viewing my practice through the lens of ‘data’ in the first instance (not something I had done before) has given me new directions for future compositions.

The Network has also been beneficial to the wider PRiSM Team, many of whom attended the first group Workshop in Spring 2022 at the University of York. Focused around ‘Improvisation and AI’, conversations over this weekend were important for the subsequent establishment of PRiSM’s own AI-Improvisation software development collaborations with Oded Ben-Tal (Adversarial Timbre Composition, 2022) and George Lewis (Forager, 2022), leading on to our current software development project with Hongshuo Fan (PRiSM Music Gesture Recognition). We are also developing future work with Franziska Schroeder and the Performance without Barriers Research Group, Queen’s University Belfast, after meeting through the Network.

At Workshop 2 (KTH, Sweden) in Autumn 2022, I was introduced to Erik Natanael Gustafsson who I then invited to join a collaboration with Zubin Kanga (Cyborg Soloists, Royal Holloway University of London) resulting in a new work for piano and multimedia that explores brain data: DEVIANCE (Howard, Ma, Gustafsson, 2022).

The RNCM has felt the effect of the Network through recent research seminars by Network members Elaine Chew and Bob Sturm, who have both become members of PRiSM’s International Advisory Board.”

Professor Robin Laney (Network Partner)

Professor of Computational Musicology, The Open University

“My team works in two general problem areas: i) computational musicology where we use principled approaches to the statistical generation of music to explore music theories empirically and ii) developing creative systems for use by novices. Interacting with the other members of the network has given me a number of new perspectives. In particular, I have valued contact with musicians using AI and data for compositional tasks and in live performance. In my own work, adherence to the style of a corpus is often important. The requirements for live performance are different and vary depending on the aesthetic goals of the performers. In particular the ability of AI agents to use data in ways that inspire reactions from human performers is interesting. I was also struck by the variety of uses of the technology for composition and performance and the way data can be used in many ways as part of artistic endeavours.”

Professor Nicola Dibben (Network Partner)

Professor in Music, and Faculty Director of Research and Innovation, University of Sheffield

“Participating in this network has provided me with a broader perspective on ‘music data’, traversing musical research domains — from the visceral experience of data in music-making through to live visualisation of performer physiological data and to questions of computational creativity.

As a researcher in music cognition, I am interested in the experience and phenomenology of music listening. This has included interpreting physiological data within constructivist models of emotional experiences with EDM and which place the socio-cultural at the core of psychological theories of music emotion. Such laboratory-based uses of music data sit alongside innovations in research at scale and in live contexts of listening, such as the RITMO MusicLab. In both cases, data can be seen as partial representations which help construct the object of study.

Simultaneously, as a popular music scholar collaborating with musicians and computer scientists on AI music generation, the network has underscored the importance of creative practice as an investigative tool.

The DataSense network has prompted a fresh set of questions for me about creative practice as an avenue for inquiry — how creative arts methodologies facilitate exploration of data and can help us reflect on the epistemic status of data.

This network experience has reinforced the need to position musicians not merely as consumers of technology but as integral partners in its development. It underscores the multidimensionality of music as a cognitive, affective, and embodied experience, urging us to utilise musical practice in performing and composing with data to navigate the challenges and use the opportunities that lie ahead.”

Professor Elaine Chew (Network Partner)

Professor of Engineering, King’s College, London

“I am deeply grateful to the illustrious leaders of “Datasounds, datasets, and datasense,” Oded Ben-Tal and Federico Reuben, for bringing together a cross-cutting collection of strong artistic and scientific researchers for scintillating idea exchanges, workshops, and outreach activities. The virtual seminar series drawing from researchers worldwide, Music Data Live at the Cheltenham Science Festival and the ensuing documentary film, the roadmap article grappling with the meaning of data, and the personal exchanges between partners across institutions have started ripples whose effects will be felt for years to come.”

Professor Bob Sturm (Network Partner)

Associate Professor of Computer Science, KTH Royal Institute of Technology, Sweden

“Being a member of the ‘Datasounds, datasets and datasense research network’ has resulted in several concrete and beneficial outcomes. For one, it has brought me and my research group into contact with interesting researchers in the arts from the UK and elsewhere. Members of the network contributed talks and performances to a festival on AI and Music I organized in Stockholm, Nov 2022 (https://musaiclab.wordpress.com/musaic-festival-2022/). One of the keynote speakers at the first workshop, Örjan de Manzano, is currently collaborating on a master’s project at my institution. I have also recently visited the institution of another network member (Emily Howard at RNCM) to give a seminar and participate in discussions with their students. Work continues between members of the network on writing an article about what ‘data’ means to each of us and our work, and how it reflects our values and priorities. In my opinion, the research network has accomplished the goals it set out, and has helped me as a researcher in music and data.”