Jev is the only sub-second model that will tell you it doesn't know

Last updated: September 18, 2026

TypeSafe shipped Jev this week, a model that returns typed judgments and calibrated probabilities instead of text. I got access and ran it against fifteen other models on the same 150 passages, plus two more tasks on top of that.

Several models beat it on accuracy and one beat it on calibration, and I will get to both, because the reason they beat it is the whole argument.

In the tier where you would actually deploy a cheap gate, under a second, Jev was the best calibrated model I measured and the only one that reliably admits uncertainty. Across the whole field of sixteen, only four models did that at all, and Jev is the only one of the four that finishes in under a second.

What Jev is

You send it state and typed questions. It returns answers with probabilities. No prose, no reasoning trace, no chat. Choice picks one option from a set, Score rates against ordered levels, Noul returns the probability a yes/no condition holds.

Independent questions ride in one request over one shared state and run in parallel, so a twelve-check gate is one call and the state is billed once.

The measurement

150 passages drawn from 35 blog posts that exist in two drafts, one rewritten into a warmer practitioner voice. Given a passage, return the probability it came from the rewritten draft. Only passages unique to one draft were used, so the answer is never in both.

That is a Noul question: a raw probability, which is the only honest way to measure calibration. Sixteen models, identical inputs, zero errors across 2,400 calls.

Expected calibration error, ECE below, is the average gap between what a model claimed and what turned out to be true, weighted by how many rows landed in each bin. Lower is better. "Unsure" is the share of rows a model placed in the ambiguous middle, between 0.35 and 0.65.

Model	Accuracy	ECE	Unsure	p50 latency	Distinct values	Cost / 150 rows
z-ai/glm-5.3-flash	74.0%	0.089	6.0%	7,527ms	28	$0.026
Claude Sonnet 5	71.3%	0.062	41.3%	1,674ms	18	$0.160
Claude Fable 5.1	70.0%	0.115	36.7%	3,293ms	19	$0.806
gpt-5.6-terra	70.0%	0.173	4.0%	1,551ms	34	not published
moonshotai/kimi-k2.5	70.0%	0.199	0.7%	12,268ms	17	$0.363
deepseek/deepseek-v4-flash	69.3%	0.183	3.3%	1,361ms	15	$0.003
gpt-5.4-mini	67.3%	0.192	6.7%	934ms	47	not published
gpt-5.6-luna	66.7%	0.243	1.3%	1,186ms	29	not published
Claude Haiku 4.5	66.0%	0.122	2.7%	631ms	11	$0.064
Jev	66.0%	0.121	34.7%	455ms	56	not published
deepseek/deepseek-v4-pro	65.3%	0.262	0.7%	11,156ms	23	$0.277
gpt-5.5	65.3%	0.190	11.3%	1,136ms	39	not published
z-ai/glm-5.3	64.7%	0.225	6.0%	4,082ms	28	$0.268
gpt-5.6-sol	64.7%	0.235	7.3%	2,452ms	33	not published
Claude Opus 5 (effort low)	64.7%	0.163	23.3%	2,090ms	33	$0.400
moonshotai/kimi-k3	58.0%	0.309	1.3%	5,677ms	30	$0.731

Costs are measured against published per-token rates. The GPT-5 rows say "not published" because I have no rate card I can cite for them, and an empty cell beats a guess. Open-weight models ran through OpenRouter, so their latency includes a routing hop and is not a clean measure of the model alone.

"Distinct values" counts how many different probabilities each model produced across 150 rows. A model emitting 11 of them is not measuring anything finer than 11 steps, whatever the decimal suggests.

Two things jump out before the calibration argument. deepseek-v4-flash did the whole run for $0.003, twenty times cheaper than Haiku, at higher accuracy. And kimi-k3, the most expensive model here at $0.731, came last on accuracy and last on calibration.

Four models out of sixteen

Sort that table by the "unsure" column and it collapses into two groups.

Model	Unsure	p50
Claude Sonnet 5	41.3%	1,674ms
Claude Fable 5.1	36.7%	3,293ms
Jev	34.7%	455ms
Claude Opus 5 (effort low)	23.3%	2,090ms
everything else	7.3% and below

Four models out of sixteen will place a meaningful share of answers in the middle of the scale. Three of them are Anthropic frontier models. The fourth is Jev.

Below that line the drop is steep and it does not care about price or vintage. Every GPT-5 variant is at 11.3% or less. Every open-weight model is at 6.0% or less. kimi-k2.5 and deepseek-v4-pro both sit at 0.7%, which is one row in 150.

Jev is the only model in the uncertainty-expressing group that runs in under a second. The next fastest is Sonnet 5, at 3.7x the latency.

The model that beat it

Sonnet 5 posted an ECE of 0.062, roughly half Jev's 0.121, with better accuracy and more willingness to sit in the middle. On the metric TypeSafe's entire pitch rests on, a general-purpose frontier model won.

It took 1,674ms to do it. That is 3.7x Jev's median, and it cost $0.16 for 150 rows against Haiku's $0.064.

This matters because of what a System One model is for. You do not put a 1.7-second model in front of 100% of traffic. The gate exists to be cheap enough to run on everything, so the expensive model only sees what survives. A 1.7-second gate in front of a 2-second model is not a gate, it is a tax.

So the comparison that decides anything is inside the latency budget where this pattern works at all.

Inside the sub-second tier

Three models came in under a second.

Model	p50	ECE	Unsure	Distinct values
Jev	455ms	0.121	34.7%	56
Claude Haiku 4.5	631ms	0.122	2.7%	11
gpt-5.4-mini	934ms	0.192	6.7%	47

Jev is the fastest model in the entire field of sixteen, the best calibrated of the three fast ones, and it flags uncertainty on 34.7% of rows against Haiku's 2.7% and gpt-5.4-mini's 6.7%.

That last column is the one that changes what your code can do.

Why "unsure" is the number that matters

Haiku and Jev tie on accuracy at 66.0% and tie on calibration error at 0.122 against 0.121. Two identical-looking models, until you look at where the probabilities land.

Haiku emitted 11 distinct probability values across 150 rows and put almost nothing in the middle. Its answers are bimodal: probably not, or probably yes. Jev emitted 56 and spread them across the whole scale.

An escalation rule is the reason to care. If your code routes uncertain rows to a human or to a bigger model, that rule fires on 34.7% of Jev's rows and on 2.7% of Haiku's. The ambiguous rows have not gone anywhere. Haiku is answering them at high confidence and sending them straight through, and you will never see them.

Every GPT-5 variant behaves the same way, and so does every open-weight model I ran. gpt-5.6-luna placed 1.3% of rows in the middle; kimi-k2.5 and deepseek-v4-pro placed 0.7%, which is one row in 150. Uncertainty is not something these models are built to report, and asking for a confidence number does not produce one, it produces a round number that looks like one.

Two more tasks

The prose task is one shape of judgment, so I ran two others against Jev and Haiku.

Task A, business category. 149 rows from a New Mexico business directory, 6 categories plus a no-match option, given a company name and city. Labels come from the directory.

Task B, commit type. 100 commits from a working repo, 5 conventional-commit types, given the subject with the type prefix stripped. The label is exact, because the prefix is the ground truth.

Task	Rows	Jev accuracy	Haiku accuracy	Jev p50	Haiku p50	Speedup
A. business category	149	79.9%	83.2%	432ms	702ms	1.62x
B. commit type	100	50.0%	42.0%	468ms	695ms	1.49x
C. prose voice	150	66.0%	66.0%	455ms	631ms	1.39x

Accuracy splits one each with a tie. Latency does not split, and Jev takes all three.

Both models score badly on task B, which says more about the task than about them. In a repo where documentation is much of the output, docs, chore and feat genuinely overlap. A person would score badly too.

The willingness gap held across both Choice tasks. Pooling them, 249 rows:

	Rows below 0.60 confidence	Distinct confidence values
Jev	41 of 249 (16.5%)	63
Claude Haiku 4.5	3 of 249 (1.2%)	12

Pooled calibration on those same 249 rows climbs for both, which is worth saying plainly because it means Haiku is not broken:

Stated confidence	Jev n	Jev accuracy	Haiku n	Haiku accuracy
0.00–0.60	41	34.1%	3	33.3%
0.60–0.80	35	42.9%	55	34.5%
0.80–0.95	51	62.7%	71	56.3%
0.95–1.00	122	88.5%	120	88.3%

Both curves rise. The difference is not that Haiku's number is wrong, it is that Haiku almost never puts a row in the top two rows of that table, so the part of the scale you would build a gate on is empty.

The labels were worse than the models

27 of Jev's 30 misses on the business task were none_of_these, and reading them, most were the directory being wrong. "Quality Inn" is filed as a travel agent. It is a hotel. "USDA Rural Development" is filed as a mortgage broker. It is a federal agency.

Two of the 30 looked like real model errors, the Santa Fe New Mexican and the Deming Headlight, both actual newspapers. Both came back at low confidence, 0.71 and 0.55. The system flagged the rows it got wrong.

I nearly ran the whole thing against a different column in that dataset, labeled valid / bad_syntax / no_mx. Every one of those labels is produced by a DNS lookup and a regex. Measuring a judgment model against mechanical labels would have produced a confident, meaningless number.

What I could not check

TypeSafe claims 40 to 1,000x cheaper and 20 to 200x faster. Neither is verifiable from where I sit.

There is no pricing page in their docs. /pricing and /limits both return 404 and no rate appears in the API reference, so the cost claim is currently untestable. On speed I measured 1.4x to 3.7x depending on what you compare against, which is a real advantage and a long way from 20x.

Jev also used more tokens than Haiku on every task, on both sides. Returning a full probability distribution costs output tokens that returning one word does not.

What I changed in my own setup

Three branches, now a rule instead of an instinct:

1. A regex, a DNS lookup or a database query can answer it. Write the code.
2. It needs multi-step reasoning, domain knowledge or generated prose. Frontier model.
3. A sensible person answers it in under a second from text you can show them. System One band.

Inside band three, one question decides it. Does your code branch on the confidence value? If not, use whatever you already have, because at 66% accuracy Jev and Haiku are the same model. If yes, 34.7% against 2.7% is the entire argument, and no amount of prompt engineering closes it.

Then I wrote a Claude Code hook so I stop having to remember. It fires on Write and Edit, only on .ts, .tsx, .mts, .js, .mjs and .py, and only when the content holds an LLM SDK call or a declaration named like a classifier. It injects the three-way test and never blocks. It is at github.com/wotai-dev/typesafe-jev-tools.

One bug in it is worth stealing the fix for. The first version read only tool_input.new_string, passed every piped check, and did nothing useful. On an Edit, new_string is only the replacement text, so changing two lines inside an existing classifier never contains the word "classify." It fired on every edit and stayed silent on exactly the case it existed for. It reads the file on disk now:

if [ -f "$path" ]; then
  content="$content
$(head -c 200000 "$path" 2>/dev/null)"
fi

If you have written a Write|Edit hook that inspects content, you probably have this bug.

Frequently asked questions

Which model was best overall?

Claude Sonnet 5, on calibration error (0.062) and accuracy (71.3%). It took 1,674ms per row to get there, which is 3.7x Jev's median. If your budget allows a 1.7-second gate, Sonnet 5 is the better instrument. Most gates cannot.

Why is Jev worth using if Sonnet 5 is better calibrated?

Because a gate has to be cheap enough to run on everything. Inside the sub-second tier, Jev had the best calibration error of the three models that qualified and flagged uncertainty on 34.7% of rows against 2.7% and 6.7%. It was also the fastest model in the whole field of sixteen, and the only one of the four uncertainty-expressing models that runs in under a second.

Can I just ask Claude or GPT for a confidence number instead?

You can ask, and you will get a number. Across 150 rows Haiku 4.5 emitted 11 distinct probability values and placed 2.7% of rows in the ambiguous middle. Every GPT-5 variant and every open-weight model behaved the same way, with kimi-k2.5 and deepseek-v4-pro at 0.7%. The number comes back; it just does not carry information you can threshold on.

Was Jev more accurate?

Not consistently, and that is not the claim. Across three tasks it lost one, won one and tied one: 79.9% against 83.2% on business categories, 50.0% against 42.0% on commit types, 66.0% each on prose voice. Accuracy is the thing you can already buy.

How much does Jev cost?

TypeSafe has not published a rate. Their docs have no pricing page and the API reference states no price. For reference, the same 150 rows cost $0.064 on Haiku 4.5 and $0.160 on Sonnet 5 at published rates.

How fast is it really?

455ms at the median, the fastest of sixteen models measured. That is 1.4x faster than Haiku 4.5 and 3.7x faster than Sonnet 5. TypeSafe's landing page claims 20 to 200x, and I did not find the workload where that appears.

---

Building with agents on real work? The WotAI community is where this kind of thing gets argued about before it turns into a blog post. The newsletter carries one every Friday.