Meet NeuroVFM: A New Neuroimaging Foundation Model Trained With Vol-JEPA on Uncurated Clinical MRI and CT Volumes

Frontier fashions be taught principally from public web information. However, scientific neuroimaging not often seems there, as a result of MRI and CT scans comprise identifiable facial options. Consequently, normal fashions underperform on brain-imaging duties. A University of Michigan analysis workforce addresses this hole with NeuroVFM, published in Nature Medicine.

What is NeuroVFM?

At its core, NeuroVFM is a generalist visible basis mannequin for neuroimaging. Specifically, it was skilled on 5.24 million scientific MRI and CT volumes. These got here from 566,915 research within the UM-NeuroImages dataset. That information spans over twenty years of routine care at Michigan Medicine.

The analysis workforce name their method ‘well being system studying.’ In quick, the mannequin learns from uncurated information generated throughout regular scientific operations. Therefore, it avoids the bottleneck of paired radiology reviews. It additionally avoids the disease-specific curation utilized in slim classifiers.

Notably, the bottom mannequin is named Vol-JEPA. It extends the sooner I-JEPA and V-JEPA strategies to volumetric medical photos. This displays a wider pattern: JEPA-style studying is increasing into medical imaging.

How Vol-JEPA Works?

Vol-JEPA is a self-supervised, vision-only algorithm. Rather than reconstructing pixels, it predicts representations in a discovered latent house. As a end result, it wants no labels, no report textual content, and no voxel decoder.

First, every 3D quantity is tokenized into non-overlapping 4×16×16-voxel patches. Next, the amount is cut up right into a small seen context and a bigger masked goal. A scholar encoder then processes the context patches.

Meanwhile, a predictor combines context latents with goal place encodings. It predicts the masked-region latents. A trainer encoder generates the ground-truth goal latents. This trainer is an exponential shifting common (EMA) of the scholar. Training minimizes a clean L1 loss between predicted and trainer latents, with gradients stopped by the trainer.

Importantly, masking is foreground-focused, utilizing precomputed head masks. Context ratios are 25% for MRI and 20% for CT, with 20% patch dropout. This design encourages the encoder to mannequin shared neuroanatomy relatively than background shortcuts.

Play</button>
<button id=”step”>Step</button>
<button id=”reset”>Reset</button>
<span class=”toggle” id=”mod”>
<button class=”sel” data-m=”mri”>MRI · 25% context</button>
<button data-m=”ct”>CT · 20% context</button>
</span>
</div>
<div class=”meta” id=”meta”>Modality: MRI. Context sampling ratio: 25%. Patch dropout: 20%. Patches are 4&instances;16&instances;16 voxels.</div>

<div class=”foot”>
<span>Illustrative animation of the strategy in “Health system studying permits generalist neuroimaging fashions” (Nature Medicine, 2026). Not an actual mannequin run.</span>
<span>Interactive explainer by <b>Marktechpost</b></span>
</div>
</div>

<script>
(perform(){
var N=12, whole=N*N;
var grid=doc.getElementById(‘grid’);
var cells=[];
// tough head-mask footprint: an ellipse of foreground patches
var fg=[];
for(var r=0;r<N;r++){for(var c=0;c<N;c++){
var dx=(c-(N-1)/2)/(N/2), dy=(r-(N-1)/2)/(N/2);
var inside=(dx*dx*1.05+dy*dy)<0.92;
var cell=doc.createElement(‘div’); cell.className=’cell’;
cell.dataset.fg=inside?’1′:’0′; grid.appendChild(cell); cells.push(cell);
if(inside) fg.push(cells.length-1);
}}

var modality=’mri’;
var ctxRatio={mri:0.25, ct:0.20};
var ctxSet=[], tgtSet=[];

perform shuffle(a){for(var i=a.length-1;i>0;i–){var j=Math.flooring(Math.random()*(i+1));var t=a[i];a[i]=a[j];a[j]=t;}return a;}

perform buildSplit(){
var pool=shuffle(fg.slice());
var nCtx=Math.max(3,Math.spherical(pool.size*ctxRatio[modality]));
ctxSet=pool.slice(0,nCtx);
// 20% patch dropout from the remainder -> nonetheless targets; right here all remaining foreground are targets
tgtSet=pool.slice(nCtx);
}

perform clearClasses(){cells.forEach(perform(c){c.className=’cell’;});}
perform paintFg(){cells.forEach(perform(c){if(c.dataset.fg===’0′)c.className=’cell’;});}

var packing containers={s:doc.getElementById(‘bStudent’),p:doc.getElementById(‘bPred’),t:doc.getElementById(‘bTeacher’),l:doc.getElementById(‘bLoss’)};
perform boxesOff(){Object.keys(packing containers).forEach(perform(ok){packing containers[k].classList.take away(‘on’);});}

var word=doc.getElementById(‘word’);
var drugs=[].slice.name(doc.querySelectorAll(‘.capsule’));
perform setStep(s){drugs.forEach(perform(p){p.classList.toggle(‘energetic’, +p.dataset.s===s);});}

var lossval=doc.getElementById(‘lossval’), lossfill=doc.getElementById(‘lossfill’);
var loss=1.00;
perform replaceLoss(){lossval.textContent=loss.toFixed(2); lossfill.type.width=Math.max(6,loss*100)+’%’;}

var stage=0;
perform render(){
clearClasses(); boxesOff();
change(stage){
case 0: // tokenize
setStep(0);
cells.forEach(perform(c){ if(c.dataset.fg===’1′){ c.classList.add(‘pulse’); setTimeout(perform(cc){return perform(){cc.classList.take away(‘pulse’);};}(c), 250);} });
word.innerHTML='<b>Tokenize.</b> Each 3D quantity is cut up into non-overlapping 4&instances;16&instances;16-voxel patches. Only foreground (head) patches are saved.’;
break;
case 1: // context/goal cut up
setStep(1); buildSplit();
ctxSet.forEach(perform(i){cells[i].classList.add(‘ctx’);});
tgtSet.forEach(perform(i){cells[i].classList.add(‘tgt’);});
word.innerHTML='<b>Context / Target.</b> A small seen <b>context</b> (blue) is sampled. The bigger masked <b>goal</b> (amber) is predicted.’;
break;
case 2: // scholar encodes context
setStep(2);
ctxSet.forEach(perform(i){cells[i].classList.add(‘ctx’,’pulse’);});
tgtSet.forEach(perform(i){cells[i].classList.add(‘tgt’);});
packing containers.s.classList.add(‘on’);
word.innerHTML='<b>Student encodes.</b> The scholar encoder E&theta; turns seen context patches into context latents.’;
break;
case 3: // predict goal latents
setStep(3);
ctxSet.forEach(perform(i){cells[i].classList.add(‘ctx’);});
tgtSet.forEach(perform(i){cells[i].classList.add(‘tgt’,’pulse’);});
packing containers.s.classList.add(‘on’); packing containers.p.classList.add(‘on’);
word.innerHTML='<b>Predict latents.</b> The predictor P&phi; makes use of context latents plus goal positions to foretell masked-region latents.’;
break;
case 4: // trainer generates goal
setStep(4);
cells.forEach(perform(c){if(c.dataset.fg===’1′)c.classList.add(‘ctx’);});
tgtSet.forEach(perform(i){cells[i].classList.take away(‘ctx’);cells[i].classList.add(‘tgt’,’stuffed’);});
packing containers.t.classList.add(‘on’);
word.innerHTML='<b>Teacher (EMA).</b> A trainer encoder, an exponential shifting common of the scholar, produces ground-truth goal latents. Gradients are stopped right here.’;
break;
case 5: // loss
setStep(5);
ctxSet.forEach(perform(i){cells[i].classList.add(‘ctx’);});
tgtSet.forEach(perform(i){cells[i].classList.add(‘tgt’,’stuffed’);});
packing containers.l.classList.add(‘on’);
loss=Math.max(0.05, loss*0.82);
replaceLoss();
word.innerHTML='<b>Smooth L1 loss.</b> Training minimizes the gap between predicted and trainer latents. Loss decreases over steps.’;
break;
}
postHeight();
}

perform subsequent(){ stage=(stage+1)%6; render(); }

var timer=null, enjoying=false;
var playBtn=doc.getElementById(‘play’);
perform play(){ enjoying=true; playBtn.textContent=’❚❚ Pause’; timer=setInterval(subsequent,1600); }
perform pause(){ enjoying=false; playBtn.textContent=’ Play’; clearInterval(timer); }
playBtn.onclick=perform(){ enjoying?pause():play(); };
doc.getElementById(‘step’).onclick=perform(){ if(enjoying)pause(); subsequent(); };
doc.getElementById(‘reset’).onclick=perform(){ if(enjoying)pause(); stage=0; loss=1.00; replaceLoss(); render(); };

var metaEl=doc.getElementById(‘meta’);
doc.getElementById(‘mod’).addEventListener(‘click on’,perform(e){
var b=e.goal.closest(‘button’); if(!b)return;
modality=b.dataset.m;
[].slice.name(this.kids).forEach(perform(x){x.classList.toggle(‘sel’,x===b);});
metaEl.innerHTML=’Modality: ‘+(modality===’mri’?’MRI’:’CT’)+’. Context sampling ratio: ‘+(ctxRatio[modality]*100)+’%. Patch dropout: 20%. Patches are 4&instances;16&instances;16 voxels.’;
if(stage>=1) render();
});

// auto-resize: publish personal offsetHeight (+40) to dad or mum
perform postHeight(){
strive{ var h=doc.physique.offsetHeight+40;
dad or mum.postMessage({sort:’neurovfm-voljepa-height’,top:h},’*’); }catch(e){}
}
window.addEventListener(‘load’,perform(){replaceLoss();render();postHeight();});
window.addEventListener(‘resize’,postHeight);
replaceLoss(); render();
})();
</script>
</physique>
</html>
“>

Performance

To measure this, the analysis workforce froze each encoder and skilled similar study-level attentive probes. The major endpoint was macro-averaged AUROC throughout 156 diagnostic duties. These cowl 74 MRI and 82 CT diagnoses.

Consequently, NeuroVFM reached 92.68 AUROC on CT and 92.49 on MRI. Moreover, it outperformed each baseline on the mixture endpoint.

Because PRIMA, HLIP, and NeuroMAE share the identical coaching information, these gaps isolate the target. In different phrases, latent prediction outperformed each report supervision and voxel reconstruction right here.

Training effectivity was additionally reported. For occasion, a full Vol-JEPA run used fewer than 1,000 GPU hours. It skilled over 7× quicker than a 3DINO baseline. It additionally match 16× bigger batches at equal reminiscence. The base encoder has 85.8M parameters; the small variant has 21.7M.

What the Model Enables?

Beyond prognosis, NeuroVFM helps a number of downstream duties. Each one reuses the identical frozen visible tokens.

• Report era: The workforce paired frozen NeuroVFM with Qwen3-14B in a LLaVA-1.5 type. This NeuroVFM-LLaVA system generated structured key findings.
• Triage: Findings handed to a reasoning mannequin produced acuity ranges: unremarkable, routine, or pressing.
• Grounded predictions: An attention-based MIL pooler maps findings to picture areas, with out region-level annotations.
• Cross-modal switch.:A CT-trained probe ran on MRI with beneath a 5-point AUROC drop.

On era and triage, NeuroVFM-LLaVA outperformed frontier baselines.

Furthermore, the analysis workforce ran a silent one-week potential research throughout the well being system (n=1,155). There, NeuroVFM reached 92.6% balanced triage accuracy versus 71.2% for GPT-5. However, its sensitivity was 86.5%, so 21 of 155 vital findings had been missed. The authors subsequently body it as determination assist, not autonomous screening.

Running NeuroVFM

In follow, utilizing the launched stack is simple. You set up the package deal, then name the pipeline helpers. The snippet under makes use of the repository’s personal API.

# git clone https://github.com/MLNeurosurg/neurovfm.git && cd neurovfm && pip set up -e .
from neurovfm.pipelines import (
    load_encoder, load_diagnostic_head, load_vlm, interpret_findings,
)

encoder, preprocessor = load_encoder("mlinslab/neurovfm-encoder")
dx_head = load_diagnostic_head("mlinslab/neurovfm-dx-ct")
generator, gen_preproc = load_vlm("mlinslab/neurovfm-llm")

# Encode a research, then predict diagnoses
batch = preprocessor.load_study("/path/to/ct/research/", modality="ct")
embeddings = encoder.embed(batch)                 # [N_tokens, 768]
predictions = dx_head.predict(embeddings, batch)  # [(label, prob, pred), ...]

# Generate preliminary findings, then optionally triage
vols = gen_preproc.load_study("/path/to/ct/research/", modality="ct")
findings = generator.generate(vols, clinical_context="LOC and nausea.")
triage = interpret_findings(findings, "LOC and nausea.", api_key="...")

Note that the stack requires FlashAttention-2 (v2.6.3) constructed from supply. Code ships beneath an MIT license. Weights use CC-BY-NC-SA-4.0, and some require entry approval with an institutional e-mail.

Strengths and Limitations

Taken collectively, the analysis paper reviews benefits alongside constraints. Both are grounded in its personal evaluations.

Strengths

• Learns from uncurated scans, with out report or label supervision.
• Shares one latent house throughout CT and MRI.
• Report era was reported as >24× cheaper and >23× much less carbon-intensive than GPT-5.
• Performance held throughout producers, subject strengths, and demographic subgroups.

Limitations

• 86.5% triage sensitivity means actual critical-finding misses stay.
• Weights are non-commercial, and the mannequin is just not FDA-approved.
• The mannequin is inclined to dataset, structure, and goal bias.
• Results come from a single educational well being system.

Check out the Paper and Repo. Also, be at liberty to observe us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The publish Meet NeuroVFM: A New Neuroimaging Foundation Model Trained With Vol-JEPA on Uncurated Clinical MRI and CT Volumes appeared first on MarkTechPost.