An evaluation framework for assessing generative AI tools in creative practice by examining situated observation of production workflows rather than just final outputs, developed through my PhD research and validated with professional music producers at UiT, UiO, and NTNU faculties.
What It Is
SCOPE is an evaluation framework developed throughout my PhD to assess AI music tools by examining creative workflows rather than just final outputs. Traditional metrics like Fréchet Audio Distance (FAD) measure statistical fit or perceptual quality, but they miss how generative systems actually affect production work. SCOPE shifts the question from “how good is the generated audio?” to “how effectively does this system support creative practice?”
The framework operates through a three-phase lifecycle. First, a system overview establishes baseline understanding of architecture and capabilities before hands-on use. Second, situated observation documents extended creative engagement through real-time field notes and post-session reflective journals. SCOPE deliberately defers formal scoring to prevent early impressions from crystallizing. Third, formal assessment synthesizes documented observations into structured scoring, with every rating linked to related evidence. This allows to trail back the existing scores to what actually happened during use and reduce “confirmation bias”. This lifecycle enforces a core design principle: observe before judging.
Evaluation criteria are organized into dual levels. System-level attributes characterize design decisions (architecture, interface modality, hardware requirements). Performance dimensions capture in-use experience across technical performance (usability, speed, audio quality), creative control (semantic alignment between intent and output, granularity of parameter control), and workflow integration (compatibility with DAWs and standard tools, fit with iterative, non-linear creative processes). This separation distinguishes what a system is from how it performs under real conditions.
The framework is supported by an open-source toolkit—REACT-frontend, Python-backend, Jupyter Notebooks, documentation and scoring templates, and session logging protocols—so researchers, developers, and practitioners can adopt the approach without building infrastructure from scratch.
Validation Study
The current project extends earlier PhD validation, which applied SCOPE over a year to four open-source systems (MusicGen, Riffusion, Magenta Studio, DDSP-VST). That long-term evaluation surfaced several patterns: text-to-audio systems fragmented production into “generate → separate → correct → process” cycles, transforming composers into curators who filter generations rather than shaping musical material. A real-time synthesis plug-in, despite simpler generative capabilities, scored highest overall through seamless workflow integration. These findings confirmed that how a system fits into a workflow can matter more than raw output quality.
Building on that foundation, the methodology was validated further through one-day studio sessions with five professional music producers from diverse backgrounds recruited by UiT, UiO, and NTNU faculties. Participants engaged in exploratory experimentation with Suno’s platform features within defined composition briefs established at session onset. The goal was to incorporate generated results into actual compositions while documenting their creative journey throughout the workflow, including real-time decision-making, iterative experimentation, and moments of discovery or friction.
Sessions ran from 09:00–16:00 with three main creative blocks (45 minutes each), breaks for rest and lunch, an assessment phase using collected materials to score against SCOPE criteria, and a final interview about overall experience. Participants used studio equipment including iMac running Pro Tools Studio and Ableton Live 12.
What This Study Achieves
The pilot operationalizes two core research questions: measuring the correspondence between user intent and system output to address embedded biases in generative tools, and documenting how AI limitations fragment production into disjointed generation, separation, and correction phases that reshape workflows during sustained creative practice.
Outcomes inform both practical utility for practitioners evaluating commercial platforms like Suno and Udio, and theoretical implications for process-centered evaluation frameworks applied to generative technologies. The anonymized dataset from N=5 participants will be released alongside adapted multi-user evaluation workflow templates so community members can apply SCOPE in their own contexts.