We used Vulcanbench Coding Agent Benchmark (Here's What We Found)

The Problem With Vendor Benchmarks

Every model provider publishes HumanEval scores. The problem: HumanEval tests autocomplete on isolated snippets. Real coding work — refactoring a retry module, implementing CSV export with edge cases, sorting a dependency graph — looks nothing like that.

When you're routing production coding tasks to a local model on a GTX 1080 Ti, you want to know: does it actually fix the bug, write quality code, and not introduce security issues?

No public benchmark answers that honestly. So we used VulcanBench.

What VulcanBench Is

52 tasks across Python, Go, TypeScript, and Rust. Each task is a real repo with a failing test suite. The agent gets the repo, the failing tests, and has to make them pass — without breaking existing passing tests.

Scoring is 5-dimensional:

• Functional — do the target tests pass? (pass/fail, weighted)
• Quality — ruff lint + radon complexity + maintainability index
• Security — bandit/semgrep scan, high/medium/low findings
• Efficiency — steps taken vs task complexity budget
• Human-like — LLM judge panel scoring patch naturalness (disabled in v0.1)

Runs inside a Docker sandbox with network off. Reproducible, isolated, no cheating.

First Results: qwythos-9b-q6 on Local GPU

We ran 7 tasks today with qwythos-9b-q6 (9B parameter quantized model, running via Ollama on a GTX 1080 Ti, 11GB VRAM):

Task	Difficulty	Functional	Total
hello-world	trivial	1.00	0.9948
py-ttl-cache-expiry	easy	0.00	0.159
py-csv-export-feature	easy	0.00	0.158
py-retry-refactor	medium	0.00	0.390
py-topo-sort-cycle	medium	0.00	0.158
py-jsonpointer	hard	0.00	0.157

Honest result: 1/6 functional on real tasks. The trivial task was perfect. Everything else hit the step budget (14 steps) without producing a passing patch.

The interesting one is py-retry-refactor — functional score 0, but quality 0.91 and security 0.95. The model wrote good, clean Python. It just didn't satisfy the contract the tests required. That's a solvable problem: better task framing, more steps, or a stronger base model.

What This Tells Us

The framework works. The model selection is the next variable.

9B quantized at Q6 is pushing the limits of what fits on 11GB VRAM with context overhead. We have larger models in Ollama (qwen2.5-coder:14b, deepseek-coder-v2:16b) that we'll run next. The step budget (14) was too tight for the medium tasks — py-retry-refactor needed 32 steps just to produce its partial attempt.

Why We're Publishing This

We use AI Router in production to route tasks to the right model. VulcanBench results feed directly into that routing — if deepseek-coder-v2:16b scores 40% higher functional on medium Python tasks, it becomes the default for that task class.

Publishing the raw numbers, including failures, is the point. If you're an AI agency that only shows you demo videos and cherry-picked outputs, you don't actually know if your models work on real code.

We do. We have the data.

---