Running DeepSeek V4 Flash across all of DeepSWE on one DGX Spark

Complete Updated October 3, 2026 · all 113 of 113 tasks scored

DeepSeek V4 Flash 0731, quantized to EXL3 3-bpw and served on a single 128 GB DGX Spark, ran the entire DeepSWE 1.1 suite — 113 original, contamination-free software-engineering tasks (swe-bench-ultra source) graded by real test suites. Every task ran at max reasoning under the same mini-swe-agent scaffold, one at a time.

Fail-to-pass coverage, by task

Each bar is the share of a task’s target tests that passed. Green solved the task outright; amber fell short. Sorted by coverage; the misses cluster near the top.

By language

DeepSWE is an even split across TypeScript, Python and Go, with a small Rust and JavaScript tail.

Every scored task

Per-task detail: tests, one-shot reward, agent steps (model calls), wall-clock minutes, output tokens and decode throughput on the Spark.

Method

• Hardware. One NVIDIA DGX Spark (GB10, 128 GB unified LPDDR5X).
• Model. DeepSeek V4 Flash 0731, EXL3 3.0-bpw, REAP-pruned to 216/256 experts, served on vLLM/sparkinfer, ~384K context — following the MiaAI-Lab single-Spark recipe.
• Reasoning. Max, via chat_template_kwargs: {thinking: true, reasoning_effort: "max"} with --max-model-len 393216.
• Harness. mini-swe-agent through the datacurve pier runner, one task per invocation, one concurrent, resumable.
• Re-runs. Tasks that ended on infrastructure errors (a model-server blip, or a quantization-related format abort) were re-run once to give them a complete attempt. Only general, task-agnostic scaffold changes were made — notably raising the format-error tolerance from 3 to 10 — with no per-task hints and no access to the held-out grading tests. On re-run, one such task recovered to a full solve and one confirmed a genuine near-miss.

Cost & energy

• Compute. ~159 GPU-hours across the scored tasks, 13.9M output tokens, ~22,436 model calls (avg 85 min/task at max reasoning).
• Electricity ≈ $2.83. The Spark draws ~120–140 W at the wall under memory-bandwidth-bound decode; ~159 h works out to ~21 kWh, and Florida Power & Light's 2026 residential rate is ~13.7¢/kWh all-in. (Power draw is estimated, so treat this as a rough figure.)
• vs. a frontier API. The 13.9M output tokens alone would run ~$209 at a $15/M-token output rate; counting input context across the model calls, the API-equivalent is several hundred dollars. Run locally, the marginal cost was roughly 1% of that.
• In DeepSWE-leaderboard format (per task). Pass@1 45% · 199 steps · 123k output tokens · $0.03/task in electricity — against the leaderboard's ~$2–4/task on frontier APIs. (Single run, so no confidence interval on Pass@1.)

How it fails

Every non-solve produces a specific signal — a named failing test or a compile error. None return plausible-looking wrong output that slips past the tests.

• Named test failures. 17 tasks missed by a single test, 24 by 2–5, 16 by more, and 3 scored zero. That last one was a single missing method: the TypeScript test suite failed to compile, so no tests ran. One missing API surface zeros the whole suite, where a dynamically-typed language would give partial credit.
• Rarely runs out of room. No task hit the 1000-step budget (the most any used was 405). It finishes and submits rather than hanging — with one exception: a single long, hard task (a large Rust change) wandered ~400 steps, exhausted the model's context window, and was cut off with non-compiling code. So the limit it can hit is context length, not the step budget.
• Overconfident summaries. The agent sometimes overstates its result — one run reported the task solved while scoring zero. The tests are the arbiter, not the agent's summary.

Variance is not characterized here: tasks are scored once, and a single clean re-run moved a near-miss by ~2 tests without changing its pass/fail. Separately, ~56 attempts failed on harness / Docker build-engine stalls — not the model — and were resolved by forcing Docker's legacy builder.

Complete run — all 113 tasks scored. Three tasks scored zero because the patch failed to compile (one TypeScript, two Rust), which zeros the whole suite; regressions count only pass-to-pass tests broken on tasks that built. Steps count model calls, not tool calls. Throughput is per-task mean decode rate on a single request.