A controlled benchmarking toolkit for automatic music transcription that systematically varies acoustic factors to separate algorithmic limitations from recording artefacts.
What It Is
Automatic music transcription—converting audio recordings into symbolic scores—is difficult to evaluate fairly because most benchmarks rely on a single, fixed recording environment. When a transcription tool performs poorly, it is often unclear whether the fault lies with the algorithm or with challenging acoustic conditions like heavy reverberation or compression. Sonitra addresses this by systematically varying acoustic factors across a consistent set of reference scores. This makes it possible to separate algorithmic limitations from recording artefacts.
How It Works
The toolkit uses a modular four-stage pipeline. It starts with a digital score—a MIDI file—which gets rendered into audio using software instruments. That audio then passes through one or more transcription services to transcribe it back into symbolic notation. Finally, the original and transcribed scores are compared using established music information retrieval metrics, including note accuracy, timing, and loudness. A resynthesis metric aligns transcribed notes against the original recording to catch temporal misalignments.
Between rendering and transcription, Sonitra introduces controlled acoustic degradations—reverberation, dynamic-range compression, signal distortion—as explanatory variables. This makes it possible to measure how specific recording conditions affect each tool’s accuracy. The pipeline is built with interchangeable components, so different synthesis engines, transcription models, and comparison metrics can be swapped in as the field evolves.
Current Status
The project began in July 2026 and expected to run through December 2026. Early work focuses on implementing the pipeline, integrating the Pedalboard Python library for audio rendering and effects, and connecting to commercial and open-source transcription services—Klangio, Moises.ai, YourMT3+, and Basic Pitch among them. The MAESTRO dataset provides the initial set of reference scores. The pipeline is being designed to log results systematically so that future experiments can be reproduced with minimal setup.
What We’re Aiming For
The immediate goal is to produce a working, reusable benchmark that independent researchers can run to compare transcription tools under controlled conditions. Alongside the open-source code, the project will publish a paper on arXiv describing the methodology and initial findings. A secondary question the work will explore is whether rankings produced under controlled, condition-varied tests align with results from standard industry datasets, and which metrics best correlate with perceived transcription quality.