Liquid AI Open-Sources Antidoom: A Final Token Preference Optimization (FTPO) Method that Reduces Doom Loops in Reasoning Models

Liquid AI has launched Antidoom, an open-source methodology that targets a typical failure mode in reasoning fashions. That failure mode is the doom loop. In a doom loop, a mannequin emits a span. It then repeats that span many times. The output continues till the context window is exhausted. Small reasoning fashions are extra susceptible to this, particularly on lengthy pondering traces and exhausting issues.

On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on exhausting math and coding prompts produced repetitive loops. After Antidoom coaching, that price fell to 1.4%. Eval scores improved throughout the board, attributable completely to the diminished looping.

TL;DR

• Antidoom reduces doom loops by retraining solely the primary loop-start token.
• FTPO spreads likelihood throughout a number of coherent options, not one substitute.
• LFM2.5-2.6B looping fell 10.2% to 1.4%; Qwen3.5-4B fell 22.9% to 1%.
• The pipeline runs in just a few hours, and the total stack is open supply.

What is Antidoom?

Antidoom is a focused repair, not a broad sampling change. It finds the precise token that begins a loop. It then trains the mannequin to desire coherent options at that single place. The remainder of the distribution stays largely untouched.

The methodology adapts Antislop. It trains on chosen/rejected pairs that symbolize a single completion token. The coaching algorithm is Final Token Preference Optimization (FTPO), which is analogous to DPO.

The coaching teaches the mannequin nothing new about math or code. It clears the looping that blocked solutions the mannequin might already produce.

Anatomy of a Doom Loop

Liquid AI workforce attributes doom loops to a few mechanisms working collectively:

Mechanism 1: overtrained tokens plus uncertainty. Some tokens usually tend to be chosen in common. Well-known examples in the wild embody ‘delve’ and ‘testomony.’ Liquid AI workforce notes this will hint again to artificial knowledge in the coaching set. In reasoning traces, high-prior continuations usually embody discourse markers reminiscent of ‘Wait’ or ‘Alternatively.’ These tokens will not be inherently unhealthy. They can mark a helpful change of technique, a verification step, or a department. When the mannequin is unsure or caught, they as an alternative turn out to be engaging fallback continuations.

For an early LFM2.5-2.6B checkpoint, the most typical loop-starting tokens have been the next.

Mechanism 2: prior context reinforces the loop. Each repetition pushes each token in the span towards likelihood that Duan et al. study this in their work on round reasoning. They hyperlink it to a “V-shaped” consideration sample. They discover that semantic repetition precedes textual repetition.

Mechanism 3: grasping sampling. Reasoning fashions normally run at low temperature for steady, reproducible traces. At temperature 0, the probably token is at all times chosen. A domestically strengthened loop then has no exit. Liquid AI experiences vital looping even at temp=0.67. Lower temperatures exacerbate the issue.

How Antidoom Locates the Failure

Antidoom generates completions on a immediate combine designed to elicit looping, at low temperature. That combine ships because the LiquidAI/antidoom-mix-v1.0 dataset. A loop is detected when a piece repeats at the least 4 instances, over at the least 60 characters.

The methodology then targets the first token of the primary repeat. At that place, it takes the bottom mannequin’s top-k log-prob options. It filters brief or non-alphanumeric noise. It retains as much as 20 believable substitutes as chosen tokens.

Each coaching row is a tuple of immediate prefix, one rejected token, and a number of chosen tokens. The chosen and rejected distributions are regularised earlier than coaching. Otherwise just a few culprits like Wait, So, and the would dominate and over-suppression would degrade reasoning.

The detection rule itself is straightforward to state in code. The snippet beneath is illustrative.

# A loop = a unit repeating >=4 instances, spanning >=60 characters.
# Returns the index of the primary token of the primary repeat (the goal), else None.
def find_loop(textual content, min_repeats=4, min_chars=60):
    n = len(textual content)
    for span in vary(1, n // min_repeats + 1):
        begin = 0
        whereas begin + span * min_repeats <= n:
            unit = textual content[start:start + span]
            repeats = 1
            pos = begin + span
            whereas textual content[pos:pos + span] == unit:
                repeats += 1
                pos += span
            if repeats >= min_repeats and span * repeats >= min_chars:
                return begin + span          # first token of the primary repeat
            begin += 1
    return None

Each detected loop then turns into one coaching row. The construction is a straightforward tuple.

# One FTPO coaching row, per the put up's [prefix, rejected, chosen] format.
row = {
    "immediate": prefix_up_to_the_loop,   # textual content earlier than the primary repeat
    "rejected": " Wait",               # the only token that began the loop
    "chosen": [" So", " Since", " The", " Therefore"],  # as much as 20 options
}

Final Token Preference Optimization (FTPO)

FTPO is a preference-optimization algorithm just like DPO. A coaching pattern has a immediate, a selected continuation, and a rejected continuation. It is constructed to alter a handful of tokens, with minimal disturbance to the mannequin in any other case.

FTPO differs from DPO in 4 methods:

1. Final token coaching: It trains solely the trailing token of a sequence that is halfway by way of era.
2. Multiple chosen tokens per pattern: It spreads likelihood throughout a bunch of options, so one overtrained token shouldn’t be merely changed by one other.
3. KL-like loss in logit area: It omits the softmax and computes divergence from reference in logits, avoiding stress on unrelated tokens.
4. Two-part regularization: Chosen and rejected logits transfer extra freely, whereas the remaining vocab stays tightly constrained.

In the Antidoom implementation, the mannequin trains for one epoch with LoRA. High LoRA ranks of 128-256 gave the perfect outcomes. Training covers all consideration and MLP projections, plus lm_head. Learning charges land round 4e-6 to 2e-5.

Training makes use of early stopping on chosen_win, the share of samples the place chosen tokens beat rejected. Stopping at chosen_win=0.35 reduce doom-loop charges from 20-30% all the way down to 1-2%. Training longer tended to degrade the mannequin.

For the early LFM2.5-2.6B checkpoint, training-set era took about one hour on 8x MI325 GPUs. Training then took about one to 2 hours on a single MI325 GPU. Generation stops after amassing 20k pairs.

How Antidoom Compares to the Usual Fixes

Results

After coaching, the doom-looping price on the early LFM2.5-2.6B checkpoint dropped from 10.2% to 1.4%. Eval scores improved throughout the board, attributable completely to the discount in looping.

Liquid AI workforce additionally ran the pipeline on Qwen3.5-4B, which is understood to loop throughout reasoning. Its doom-looping price dropped from 22.9% to 1% beneath grasping sampling. Eval scores elevated markedly.

The eval rating modified inversely with the doom-loop price as temperature rose. After coaching, each fashions confirmed a efficiency drop close to temp=1.0. This is predicted, since higher-temperature sampling can favor less-preferred tokens. Once looping is eliminated, near-greedy sampling gave the strongest scores in the fashions examined.

Liquid AI workforce flags a associated level about frequent apply. The perception that greater temperatures help reasoning could also be conflated with the impact of doom-looping. In their exams, as soon as loops are gone, near-greedy sampling performs greatest.

Multiple rounds can assist. The first spherical rejects loop-causing tokens and reweights towards options. That can expose new failure factors, which a second spherical then targets.

Interactive Explainer

Use Cases with Examples

• On-device reasoning fashions: Sub-1GB reasoning fashions just like the LFM2.5 household can stall mid-proof on exhausting prompts. Antidoom recovers the accuracy these loops have been costing.
• Small coding brokers: A 4B coding mannequin can loop on a tough debugging hint and burn its context window. Removing the loop lets it attain the repair it already knew.
• Agent pipeline price management: Loops devour tokens till context exhaustion. Cutting them reduces wasted tokens and latency throughout lengthy agent runs.
• Post-training restore. Teams transport fine-tuned reasoning checkpoints can run Antidoom as a cleanup go in just a few hours.

Strengths and Challenges

Strengths:

• Targeted: it edits the primary loop token and leaves the remainder of the distribution largely intact.
• Fast: the entire pipeline runs in just a few hours.
• Measured: LFM2.5-2.6B fell 10.2% to 1.4%; Qwen3.5-4B fell 22.9% to 1%.
• Open supply: era, detection, and the FTPO coach are all launched.
• Recovers, not teaches: it restores solutions the mannequin might already produce.

Challenges:

• It can expose new failure factors, so a number of rounds are generally wanted.
• Over-training degrades the mannequin, so early stopping on chosen_win is required.
• Reported outcomes cowl LFM checkpoints and Qwen3.5-4B, each small reasoning fashions.
• Performance can drop close to temp=1.0 after coaching.
• Each mannequin wants its personal generated looping dataset.

Check out the Technical details and GitHub Repo. Also, be at liberty to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The put up Liquid AI Open-Sources Antidoom: A Final Token Preference Optimization (FTPO) Method that Reduces Doom Loops in Reasoning Models appeared first on MarkTechPost.