A research study utilizing RAVE models for live interactive electronics, investigating multi-modal control via MIDI controllers, NGIMU sensors using OSC protocol, and audio inputs through a custom MAX/MSP patch deployed in two performances.
This project was part of my research residency at NTNU, Department of Music Technology under the supervision of Professor Andreas Bergsland. The study investigated latent space representations in neural audio models to enhance artistic control within generative systems. The Realtime Audio Variational autoEncoder (RAVE) is an established method in the field of neural audio synthesis, and this study utilized RAVE for investigating different means to leverage its learned latent space without developing or modifying the original architecture.
Methodology
RAVE employs a two-stage training procedure combining representation learning with adversarial fine-tuning. We train ed several RAVE models using a Karaoke dataset across multiple architectures and configurations. To facilitate integration into live electronics workflows, a MAX/MSP patch was created for model deployment and control interface implementation. Control inputs included MIDI controllers such as Native Instruments Traktor Kontrol F1 and FaderFox UC44 Universal Controller, NGIMU sensors. We integrated the NGIMU sensors within the MAX/MSP patch using OSC protocol. We used hand movement as input signal, as moving the hands in air while holding them during performance. Additionally, we integrated audio inputs through microphones capturing speech and vocals to complete the exploratory approach to sound creation.
What This Study Achieved
The system was deployed in two live performances. The MAX/MSP patch integrated RAVE models with multiple control interfaces operating simultaneously: MIDI controllers, NGIMU sensors transmitting via OSC protocol, and microphone inputs for speech and vocals. Wikinator was incorporated into the patch to learn and parametrize the high-dimensional input vectors, which typically consisted of 12 dimensions, mapping them to lower dimensions for manageable control during performance. Dataset preparation addressed latent space continuity issues by separating audio into female and male groups, segmenting files into smaller chunks aligned with phrase boundaries, and removing unnecessary silent segments. For models trained on drum datasets, MIDI-based control produced staccato, abrupt outputs with high-attack and low-decay-sustain characteristics, limiting compositional utility. Switching to audio input control using mouth-generated vowels and sounds yielded results suitable for our compositions. The patch also included VST audio effects such as delay, reverb, and echo for additional sound shaping.
Conclusions
The two performances demonstrated that multi-modal control of RAVE latent spaces is viable for live electronics when supported by dimensionality reduction techniques. Direct manipulation of high-dimensional vectors proved impractical; mapping them to lower-dimensional parameters through Wikinator provided the necessary control resolution. The choice of control interface affected output quality in model-dependent ways: MIDI controllers worked for vocal models but produced unsatisfactory results with drum models, where audio input from mouth sounds provided better control. Dataset preprocessing—specifically separating recordings by vocal range, segmenting at phrase boundaries, and minimizing silence—was necessary to obtain a continuous latent space without gaps that would decode to silence. Among the control methods tested, NGIMU sensors using OSC protocol provided the most intuitive and engaging interaction for us (performers). Future work could extend this approach to additional model types.