Sound Made Simple - Exploring Gestural Interfaces to Make Music Creation More Accessible

ABSTRACT

This thesis investigates how intuitive, movement -based gestural interaction can lower the barriers to musical expression for non-musicians, and explores the potential of gestural interfaces to create more inclusive and playful ways of engaging with sound. Through the development of a webcam-based prototype that tracks hand movements in real time, users are able to generate and manipulate musical elements without the need for prior musical knowledge or technical skill. The system emphasizes immediacy, accessibility, and embodied interaction, turning gestures into a creative language. With a research through design approach at its core, user testing and iterative design informed the interface’s evolution, highlighting the value of removing traditional musical hierarchies in favor of more exploratory, sensory-driven experiences. The project contributes to the field of interaction design by proposing an alternative model for digital music-making—one that centers accessibility, simplicity, and joy.

Introduction

The Problem

Despite music’s deep cultural and emotional significance, the act of creating music remains largely inaccessible to those without formal training or technical expertise. Beginners, non-musicians, and children often face steep barriers, from complex software to intimidating interfaces that discourage creative exploration. There is a pressing need for intuitive, user-friendly tools that open the door to music-making for people of all backgrounds and abilities.

Personal Motivation

At the core of this thesis, lies the broader vision of leveraging technology to make creative experiences more accessible. From a personal lens, creating music has been an aspect of my life that has brought immense joy and fulfillment - and I hope to create an avenue for more people to be able to experience this joy.

Gestural Focus

The intersection of music technology and accessibility has become a growing field of interest in both academic and design communities. As digital tools for music creation continue to evolve, designers and researchers are increasingly exploring ways to make these experiences more intuitive, inclusive, and playful. Gestural interfaces — tools that translate human movement into sound or control — present a promising pathway for lowering the barriers to music-making, especially for non-musicians and individuals with limited access to traditional instruments or training. Moreover, one of the primary obstacles that deter people from pursuing music creation as a hobby is the lack of resources such as instruments, specialised equipment and lessons (Gaunt & Hallam, 2009). A gestural interface would enable us to create an experience that can be controlled simply using hands and a basic laptop - removing the need for acquiring any additional specialised equipment.

Research Approach

The project employs an iterative Research Through Design approach and use the Spiral Model as a framework for guidance - focused on evolving loops of research, prototyping, testing and refinement.

Research Question

How can gestural interfaces be configured to make music creation easier for beginners?

Literature Review

The Literature Review examined 4 areas of focus:

1. The Obstacles Associated With Music Creation

- Technical Complexity - Steep learning curves discourage beginner

- Resource Requirements - instruments, software and lessons are expensive

- Time Constraints - music learning traditionally demands extensive time commitments and discipline

2. The Benefits of Engaging in Music Creation:

- Improved Mental Health - Music has therapeutic effects and helps with anxiety

- Improved Brain Function - Enhanced cognitive abilities alongside neurogenesis and neuroplasticity

- Improved Focus and Concentration

- Personal Fulfillment and Improved Overall Wellbeing

3. The Relationship Between Gestures and Music

- Conducting Gestures in Western Classical Music - Used to convey tempo, dynamics and expressive nuances to orchestral performers

- Gestural Communication in South Asian Classical Music - Mirror melodic contours and emphasize rhythmic cycles

- Gestural Interfaces in Contemporary Music Technology - Manipulation of electronic sounds via movement - expands the expressive capabilities of performers

4. State of The Art: The Current Landscape of Music Creation Tools

- Professional Digital Audio Workstations - very capable but highly complex professional tools

- Casual and Gamified Music Production Apps - Beginner Friendly with Simple Interfaces

- Tactile and Physical Music Making tools - ranging from beginners toys to advanced sequencers

- Generative AI - music with text prompts

- Experimental and Interactive Research Projects - such as art installations

Case Studies

In the case studies, I examined 3 gestural music making tools in depth:

1. Theremin by Leon Theremin

An Electronic musical instrument from the 1920s played without physical contact by moving the hands near two antennas that control pitch and volume. Simple interface but extremely difficult to master.

2.MiMu Gloves by Imogen Heap

Wearable musical controllers that translate hand gestures and movements into sound and music control, enabling expressive, touchless performance. Highly capable but very complex to use and set up.

3. MotionComposer by Robert Wechsler

A camera based interactive system that translates body movements into music, designed especially for people of all abilities to create sound through motion. Very easy to use but incredibly expensive 16,000 EUR.

Key Findings

1. Non-contact gestural interfaces seem to be well suited for be- ginners.

Low friction and equipment based hassle - even cognitively im- paired individuals can use products like the motioncomposer.

2. Ease of use comes at the cost of expressiveness and granular control

We can see that the ease of use generally goes down when aspects of expressiveness and granular control are prioritised. It raises the question of how important versatility and precise control is for be- ginners and non-musicians, and whether other aspects such as ease of use should be prioritised at the cost of granular control.

Opportunities and Gaps

All these projects utilize specialised proprietary hardware. We can remove specialized proprietary hardware, remove the cost barrier by making it free and simplify the setup complexity and improve ease of use.

Design Process & User Testing

The core methodologies used were Research through Design alongside Qualitative Research through interviews and discussions. The aim was to begin user testing as early as possible and allow the project to evolve iteratively with feedback from users - taking into account their suggestions, pain-points and a wishlist of features that they would like to see integrated into the experience.

Initial Idea

The initial idea began with the intention to create a simple camera based interface that would track the user's movements or hands as input, and produce music as the output. It should not require any specialised equipment and should work on any basic laptop as long as it has a webcam.

The concept for the first prototype (illustrated above) tracks the user's hands on the X and Y axis and the position of their hands determines what music is played. I split the X axis into 7 colums, each of which would trigger a chord from the C major scale - the 'diatonic' nature of the selected chords would ensure that the music does not sound dissonant or jarring. Similar to most conventional instruments, moving towards the left would trigger 'lower' notes with more bass and moving right would trigger higher notes. The Y axis would determine how frequently each chord is triggered - a higher position would make the chords trigger more frequently while a lower position would trigger at a slower rate.

Development

The prototypes were developed using React alongside AI Coding tools including ChatGPT, Claude and Google Gemini. There was significant trial and error involved considering the novel nature of the interface. The coding environment used was CodeSandBox and the sounds were designed using Ableton Live.

A total of 5 iterative prototypes were created after extensive user testing, discussions and interviews with the users. Their feedback on pain points was gathered and a wishlist of features was also collected and progressively implemented into the prototypes.

Other Explorations

At various stages of the design phase, I also embarked on tangents to explore more ideas for gestural music making tools, namely a gestural step sequencer to enable people to experience sequencing and composition in a simple way and secondly the 'drawing harp' which was intended to be a drawing tool that makes music with the user's strokes. A gestural drummer was also created and implemented into the final app.

Design Outcome

The project concluded with the creation of the Gestural Music Maker. The video below demonstrates the app.

Public Installation

Gestural Music Maker Installation at Minibooth (FEZ Berlin)

Minibooth is an annual event for kids hosted at FEZ Berlin where music making tools are showcased for the attendees to interact with. I was invited to showcase the Gestural Music Maker which received significant attention and positive feedback. It was a great opportunity to test the experience with children and parents. It was a joy to see them having a great time.