Can a fast decision model tell a correct patch from a broken one?
I asked TypeSafe's Jev to score 113 DeepSWE patches from my local DeepSeek run. Its scores did not separate passing patches from failing ones.
October 4, 2026
My local DeepSeek V4 Flash run on DeepSWE 1.1 solved 51 of 113 tasks (results). Many of the other 62 missed by a test or two. I wanted to know if a cheap external judge could flag those patches before submission so the agent would keep working.
I tested Jev, TypeSafe’s structured-decision model. It answers questions with calibrated probabilities instead of text. I gave it each task’s spec and the agent’s final patch and compared its scores with the actual test results. It did not separate passing patches from failing ones. The experiment cost about $0.10.
Setup
- Data: for each of the 113 tasks, the issue text, the agent’s patch, and whether it passed.
- Holistic test: one question per task, “Is this patch a correct, complete solution?” Jev returns a probability from 0 to 1.
- Per-requirement test: the local model split each spec into requirements (about 14 per task). I asked Jev “Does this patch satisfy this requirement?” for each one, then took the lowest score, the average, and the share above 0.5.
- Metric: AUC, the probability that a solved patch scores higher than an unsolved one. 0.5 is chance. I treated 0.65 as the minimum for a usable gate.
Results
| Signal | AUC |
|---|---|
| Holistic P(correct) | 0.45 |
| Lowest requirement score | 0.40 |
| Average requirement score | 0.40 |
| Share of requirements above 0.5 | 0.51 |
All four are at or below chance. The holistic scores were also less accurate than predicting the base rate for every task (Brier 0.276 vs 0.247).
The requirement scores barely differed by outcome: solved patches averaged 0.89, unsolved 0.91. One patch that failed to compile, so no tests ran at all, still scored 0.80 on its lowest requirement.
Why it didn’t work
Jev appears to check whether a patch touches the parts of the code the spec describes, not whether the code is correct. Here the failures were a missed edge case or a broken existing test, so that check can’t catch them. TypeSafe describes Jev as a model for classification, routing, and extraction that doesn’t reason about code, so this result fits its stated scope.
For patch correctness, the tests are still the only reliable signal. Jev may work for the narrower decisions it was built for, such as routing tasks or extracting fields from logs.