Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI
Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, often MIDI. Single-instrument transcription already works fairly effectively. However, transcribing a full multi-instrument combine stays tough. Kyutai and Mirelo crew now launch MuScriptor to shut that hole. It is an open-weight mannequin educated on actual, multi-instrument recordings throughout many genres.
This article explains how MuScriptor works, what the benchmarks present, and the way to run it.
What is MuScriptor?
At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a brief audio phase. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In impact, transcription turns into a language-modeling job, following the MT3 tokenization scheme.
The launch ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and massive (1.4B). The inference code makes use of the MIT license. The weights use CC BY-NC 4.0, so business use is restricted.
How the Three-Stage Pipeline Works
MuScriptor’s foremost concept is knowledge, not structure. Accordingly, coaching strikes by means of three levels, and every builds on the final.
- Pre-training makes use of D<sub>Synth</sub>, roughly 1.45M MIDI information. An on-the-fly pipeline synthesizes them throughout coaching. Augmentations embody pitch shifting, tempo adjustments, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
- Fine-tuning makes use of D<sub>Real</sub>, an inside set of 170,000 recordings. Together they whole greater than 11,000 hours with aligned notice annotations. Most alignments come from audio-symbolic synchronization utilizing interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a most time-dilation issue.
- Reinforcement studying post-training makes use of D<sub>RL</sub>, 300 manually verified tracks. The crew applies a GRPO-like technique combining REINFORCE with group-relative benefit normalization. The reward sums three F-scores: onset, body, and offset. As a consequence, the mannequin learns to favor cleaner transcriptions.
perform pushEvent(n,form){
var pitchNames=[“C2″,”G2″,”E3″,”C4″,”A4″,”D5”];
var line=doc.createElement(“div”);line.className=”ln”;
var p=pitchNames[n[0]]||”C4″;var inst=rows[n[0]].inst;
if(form===”begin”){
line.innerHTML='<span class=”st”>NoteBegin</span> t=<span class=”pi”>’+n[1].toFixed(2)+’s</span> pitch=<span class=”pi”>’+p+'</span> inst=<span class=”in”>’+inst+'</span>’;
}else{
line.innerHTML='<span class=”en”>NoteFinish</span> t=<span class=”pi”>’+(n[1]+n[2]).toFixed(2)+’s</span> (‘+inst+’)’;
}
stream.appendChild(line);stream.scrollTop=stream.scrollHeight;
}
perform play(){
if(enjoying)return; enjoying=true;
var btn=doc.getElementById(“playBtn”);btn.disabled=true;btn.textContent=”● Transcribing…”;
playhead=0; resetStream();
var c=classify(present);
var seen=c.tp.concat(c.fp).kind(perform(a,b){return a[1]-b[1];});
render(true);
var t0=null,SPEED=DUR/4200; // ms mapping
perform body(ts){
if(!t0)t0=ts;
playhead=Math.min(DUR,(ts-t0)*SPEED);
// reveal notes + emit occasions
seen.forEach(perform(n,i){
var key=”s”+i;
if(n[1]<=playhead && !streamed[key]){streamed[key]=1;
var rects=noteLayer.querySelectorAll(“rect”);
pushEvent(n,”begin”);
setTimeout((perform(nn){return perform(){pushEvent(nn,”finish”);};})(n),120);
}
});
// replace opacities
Array.prototype.forEach.name(noteLayer.querySelectorAll(“rect”),perform(r){});
render(true);
if(playhead<DUR){raf=requestAnimationFrame(body);}
else{enjoying=false;btn.disabled=false;btn.textContent=”
Transcribe once more”;
headLine.setAttribute(“opacity”,”0″);postSize();}
}
raf=requestAnimationFrame(body);
}
// controls
doc.getElementById(“levels”).addEventListener(“click on”,perform(e){
var b=e.goal.closest(“button”);if(!b)return;
Array.prototype.forEach.name(this.kids,perform(c){c.classList.take away(“on”);});
b.classList.add(“on”);present=b.getAttribute(“data-s”);
playhead=DUR;render(false);setMetrics();postSize();
});
doc.getElementById(“condToggle”).addEventListener(“click on”,perform(){
cond=!cond;this.classList.toggle(“on”,cond);
playhead=DUR;render(false);postSize();
});
doc.getElementById(“playBtn”).addEventListener(“click on”,play);
// pipeline shimmer
var pnodes=doc.querySelectorAll(“#pipe .node”),pi=0;
setInterval(perform(){
pnodes.forEach(perform(n){n.classList.take away(“sizzling”);});
pnodes[pi].classList.add(“sizzling”);pi=(pi+1)%pnodes.size;
},900);
// resize to dad or mum (PhrasePress embed)
perform postSize(){
strive{var h=doc.physique.offsetHeight+40;
window.dad or mum.postMessage({muscriptorHeight:h},”*”);}catch(e){}
}
window.addEventListener(“resize”,perform(){render(false);postSize();});
// init
playhead=DUR;render(false);setMetrics();setTimeout(postSize,60);setTimeout(postSize,400);
})();
</script>
</physique>
</html>
“>
Performance
For analysis, the analysis crew use D<sub>Test</sub>, 372 held-out tracks with correct annotations. They report instrument-agnostic metrics from the mir_eval library. Among them, Multi F1 is strictest, because it additionally requires the proper instrument.
The desk beneath traces every coaching stage towards the YourMT3+ baseline, utilizing the big (~1.3B) mannequin.
| Model (D<sub>Test</sub>) | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
|---|---|---|---|---|---|
| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |
| MuScriptor · D<sub>Synth</sub> | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> + D<sub>RL</sub> | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |
Clearly, each stage improves outcomes, and actual knowledge issues most. Synthetic-only coaching reaches aggressive body F1 however weak onset and multi scores. Adding D<sub>Real</sub> then lifts all metrics by roughly 20 factors. Finally, RL post-training reduces false negatives and sharpens onset timing.
Cross-dataset assessments level the identical means. For instance, body F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset keep decrease on arduous kinds like chorals.
Getting Started
Installation takes one command, and inference streams notice occasions instantly.
# pip set up muscriptor (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel
# Downloads the default "medium" variant (additionally accepts "small" / "massive")
mannequin = TranscriptionModel.load_model()
# Stream notice occasions; optionally situation on recognized devices
for occasion in mannequin.transcribe("audio.wav", devices=["acoustic_piano", "drums"]):
print(occasion) # NoteBeginEvent / NoteFinishEvent / ProgressEvent
# Or write a MIDI file instantly
Path("out.mid").write_bytes(mannequin.transcribe_to_midi("audio.wav"))For the launched fashions, hold cfg_coef at 1, since they’re already RL post-trained. Additionally, uvx muscriptor serve launches a browser internet UI with a dwell piano roll.
Use Cases with Examples
Because the output is normal MIDI, many workflows open up:
- Producers can extract a MIDI bassline from a combination, then re-voice it in a DAW.
- Musicologists can convert historic recordings into editable scores for evaluation.
- MIR researchers can feed transcriptions into chord or key recognition programs.
- Educators can construct follow instruments exhibiting a dwell piano roll throughout playback.
- Developers can transcribe solely drums by passing instrument conditioning.
Strengths and Weaknesses
Strengths:
- Trained on 170k actual recordings spanning classical to heavy metallic.
- Open weights plus MIT-licensed inference code, in three dimension variants.
- Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub>Test</sub>.
- Instrument conditioning customizes output and stabilizes cross-segment predictions.
- A streaming API emits notice occasions and MIDI, alongside a browser internet UI.
Weaknesses:
- Weights are CC BY-NC 4.0, so business deployment is restricted.
- The tokenizer drops velocity and can’t characterize overlapping same-pitch, same-instrument notes.
- Onset and offset accuracy keep decrease on chorals and related kinds.
- The massive mannequin needs a GPU for sensible pace.
- The 5-second phase dimension limits long-range context and inference pace.
Check out the Paper, GitHub Repo and Model Weights. Also, be happy to observe us on Twitter and don’t neglect to be part of our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us
The put up Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI appeared first on MarkTechPost.
