Breeze Halo: a task-specific 0.8B hallucination judge that runs on CPU and outclasses its 27B parent

For a narrow, high-volume, well-defined task, a small model trained on the task beats a large general model — on precision, on speed, on cost, on privacy.

Part I — for everyone

The problem in one paragraph

Breeze Buddy runs voice agents that call customers: confirm an order, recover an abandoned cart, help a driver with their plan, verify a refund. The agents are large language models following a script. Most of the time they follow it. Sometimes they don't: an agent tells a customer their cart has "a few other products" when it has one; says "I've sent the payment link" when nothing was sent; quotes a return policy the store doesn't have; reads a pincode back with a digit wrong. We call these hallucinations, and at call-center volume even a small rate is thousands of wrong statements a day. Somebody — or something — has to listen to every call and catch them.

Why not just ask a big model?

The obvious move is to hand every transcript to a large model and ask "did the agent make anything up?". We tried it. Three problems:

1. It cries wolf. General models are suspicious of anything unfamiliar. Given a Kannada driver-support script they've never seen, they flag legitimate scripted lines as fabrications. In our benchmarks a 27-billion-parameter model flagged roughly one clean call in three. Reviewers stop trusting a tool that's wrong that often.
2. It's slow and expensive at scale. Seconds per call on a big GPU, or an API bill per call, for hundreds of thousands of calls a day.
3. The data can't leave. Customer conversations shouldn't be shipped to a third-party API to be graded.

The idea: a small model that knows exactly one job

Instead of a generalist, we trained a specialist: a 0.8-billion-parameter model — about 30× smaller than the big model above, small enough to run on a laptop CPU — that does nothing except judge these calls. It learned the rubric, the languages (Hindi, Kannada, Malayalam, Hinglish), the shape of our scripts, and the specific ways our agents go wrong.

We call it Breeze Halo.

The teaching worked like this. A larger "parent" model (Qwen3.8-27B) produced first-draft labels; a supervised verification pass corrected them (the parent alone was too noisy); the small model was trained on the corrected labels, plus deliberately injected hallucinations so it saw every type often. Then a second stage taught it from its own mistakes: we collected calls it wrongly flagged, wrote look-alike clean calls, and trained it to prefer the right answer. The model is refreshed through regular fine-tuning cycles as new agents and new failure patterns appear.

What it reads and what it says

Breeze Halo is given the same things the agent had: the agent's script, the facts it was handed (order, price, address), the tool results it saw, the actions it actually took, and the transcript. It answers with a short list: which agent turn, which kind of error (five types: a fact the customer corrected; a self-contradiction; an unauthorized promise; a claimed action that never happened; a made-up fact), the exact words, and why.

How good is it?

On a held-out benchmark of production calls — agents it was trained on plus agents it had never seen — it catches 78 of every 100 hallucinated calls and wrongly flags only 3.3% of clean ones. Put side by side with general models on the same calls and the same instructions:

The untrained small models can't do the task at all. The big parent finds slightly more but is wrong so often that every flag needs a human. A frontier API model (Claude Sonnet 5) matches Breeze Halo on F1 — it catches more but raises three times the false alarms, and every call leaves the building. The student is the only judge you can act on directly, it runs locally, and it is 8× faster than the parent it learned from.

On a harder, independent benchmark across ten agent types — including agents Halo had never seen — the same picture held: Breeze Halo scored F1 0.59 with 3.9% false alarms; Claude Sonnet 5, zero-shot with the same prompt, F1 0.49 with 11% false alarms; the 27B, 0.29 with 37%. Halo's misses were mostly new patterns (an agent saying "good afternoon" when its clock tool said 7 pm) — the kind of thing the next fine-tuning cycle absorbs.

Why this matters beyond one company

The lesson is general: for a narrow, high-volume, well-defined task, a small model trained on the task beats a large general model — on precision, on speed, on cost, on privacy — and it keeps improving with each cycle because it's cheap to retrain. That is the case for task-specific small language models at the edge, and Breeze Halo is one worked example.

---

Part II — for practitioners

1. Task formulation

Hallucination judging is cast as conditional generation: the model receives a structured context and emits a JSON object. The context has six sections — POLICY (the agent's script, summarised with verbatim allowed lines, prices and offers), FACTS (the payload handed to the agent), TOOL RESULTS (logged), ACTIONS (functions called), DATA FETCHED (functions called whose results were not logged), TRANSCRIPT (numbered turns, agent/customer). The output is {"flags":[{turn_id,type,span,evidence}]} with five types: H1 customer-corrected fact, H2 self-contradiction, H3 unauthorized commitment, H4 claimed action without a tool call, H5 ungrounded fact.

Two rubric decisions matter for precision. Verbatim scripted lines are allowed even when they sound like promises — most false alarms of general models come from this. Facts plausibly supplied by a DATA FETCHED function are unverifiable, not H5 — the platform under-logs tool results, so a judge that punishes every unlogged fact is useless. A separate stale-payload carve-out (the agent reads its facts correctly but reality has moved on) is excluded from H1.

Evaluation is at two granularities: call level (was a hallucinated call flagged?) and span level (correct turn, type and overlapping span). We report precision, recall, F1, the false-alarm rate on clean calls, and span F1.

2. Supervision pipeline

Production calls were pulled read-only from the platform's analytics API, normalised (turn segmentation, STT-echo removal, PII masking of names and phone numbers), and paired with a per-template policy distilled from the agent's system prompt.

Labels came in stages. A larger parent model (Qwen3.8-27B, served locally) produced initial labels in a single pass; its zero-shot precision was insufficient, and every flagged call went through a supervised verification pass, with later cycles using human-guided review. A held-out evaluation set was adjudicated separately and never trained on; a second held-out set removes four agent types entirely from training to measure generalisation to unseen agents.

Because real positives are sparse and skewed to one failure mode, we synthesised hallucinations on real calls: for each type, a target agent turn is rewritten to inject an error grounded in that call's facts (a wrong price, a promise absent from the script, a claimed action with no tool call), keeping the rest of the call intact. Targeted injections mirrored the six production bug patterns observed in review (cart padding, split product titles, fake "sent" claims, invented policies, transfer stalls, wrong variants). Fully synthetic novel agents (new scripts, new domains) were added to broaden the script distribution.

3. Training recipe

Base: Qwen3.5-0.8B (24 layers, hidden 1024, GQA, 262k-position RoPE), thinking mode disabled in every stage and at inference.

Stage 1 — SFT. LoRA rank 32 on all linear projections, bf16, cosine schedule with 3% warm-up, learning rate 1e-4, batch 1 × 16 accumulation, 8k-token sequences, 3 epochs. The assistant target is the bare JSON.

Stage 2 — DPO. With the SFT model as reference, preference pairs were mined from the model's own behaviour on the training distribution: (a) false alarms → chosen = clean, rejected = the model's flag; (b) labeller flags overruled in verification → same; (c) hard negatives on real scripts — clean calls rewritten to contain the two patterns that most often fool a judge (multi-option price quoting; digit read-backs with a customer correction), each paired with the decoy flag a nervous judge would raise; (d) misses → chosen = true flags; (e) true-positive anchors → chosen = true flags, rejected = clean. LoRA rank 16, β 0.1, learning rate 2e-5, 2 epochs, curriculum-ordered from anchors to near-miss negatives.

4. What moved the needle (ablations)

5. Results

Primary benchmark — held-out production calls, trained and unseen agents mixed, same prompt for every judge: see Part I table — Breeze Halo F1 0.85 vs 0.85 for Claude Sonnet 5 (P 0.94 / R 0.78 vs P 0.83 / R 0.87; false alarms 3.3% vs 10.9%; span F1 0.64 vs 0.55), 0.70 for the 27B parent and ≈0 for untrained 0.8B/2B; false alarms 3.3% vs 36% for the parent; 0.36 s vs 2.9 s per call on an RTX 5090. The 27B produced malformed JSON on ~7% of calls, the untrained small models on 4–8%; Halo on none.

Breakdown by familiarity (same benchmark): on the trained-agent portion Halo reaches precision 0.94 / recall 0.87 / F1 0.90 with 4.6% false alarms (Hindi F1 0.97, Kannada 0.60); on the never-seen portion recall is 0.48 at 1.3% false alarms — conservative by design until the agent is onboarded.

Independent ten-agent benchmark (two agents absent from training) with a frontier API competitor: Breeze Halo P 0.59 / R 0.59 / F1 0.59 / FA 3.9%; Claude Sonnet 5 zero-shot P 0.38 / R 0.71 / F1 0.49 / FA 11.1%; Qwen3.8-27B P 0.18 / R 0.82 / F1 0.29 / FA 36.7%. Halo is the highest-F1 and most precise judge; the larger models' recall edge comes with 3–9× the false alarms.

Unseen agents. In a controlled experiment that removed four agent types from training entirely, the model still found most of their hallucinations but its false-alarm rate on those agents was several times the trained-agent rate, and the DPO stage — which sharpens the model on scripts it knows — made it more suspicious of unfamiliar ones, not less. No data lever moved this; it is a coverage effect (the judge cannot know a script it has never seen), and it improves after onboarding: adding the agent's policy file and a small labelled sample to the next fine-tuning cycle brings it to trained-agent levels.

6. Edge inference

A 0.8B model with a 1.75 GB footprint is deployable on a single CPU core group with no accelerator; the cost per call is the prefill of a few thousand tokens. Naive PyTorch int8 dynamic quantisation is ~3× faster (≈4.5 s) but not usable: it degrades the judge to F1 0.21 with frequent malformed output, so it is excluded from the table. Calibrated integer quantisation (e.g. a llama.cpp/GGUF build with quantisation-aware evaluation, not yet applied) is the standard next step for laptop- and edge-class latency, and must be re-validated against the held-out set before use.

7. Limitations and ongoing work

Labels are model-produced and verification-corrected rather than human-annotated from scratch. The model is domain-bound to scripted Indic voice agents and to its six-section prompt. False alarms on un-onboarded agents and recall on novel hallucination patterns are the two active gaps; both are addressed operationally by regular fine-tuning cycles rather than by architecture. Span localisation (span F1 0.64) is adequate for review queues, not for automatic redaction.

8. Reproducibility

Base Qwen3.5-0.8B (Apache-2.0); Unsloth + TRL for SFT and DPO; transformers for inference. Inference code, the rubric, the prompt format and the onboarding recipe ship with the model. Weights are private (proprietary training data; merchant identities and scripts unmasked).

---

Breeze, Juspay — 2026.