We used Vulcanbench Coding Agent Benchmark (Here's What We Found)
We stopped trusting vendor benchmarks for local model selection. So we built VulcanBench — 52 real coding tasks, Docker sandbox, 5-metric scoring. Here's what our first runs revealed.
The Problem With Vendor Benchmarks Every model provider publishes HumanEval scores. The problem: HumanEval tests autocomplete on isolated snippets. Real coding work — refactoring a retry module, implementing CSV export with edge cases, sorting a dependency graph — looks nothing like that. When you're routing production coding tasks to a local model on a GTX 1080 Ti, you want to know: does it actually fix the bug, write quality code, and not introduce security issues? No public benchmark answers that honestly. So we used VulcanBench. What VulcanBench Is 52 tasks across Python, Go, TypeScript, and Rust. Each task is a real repo with a failing test suite. The agent gets the repo, the failing tests, and has to make them pass — without breaking existing passing tests. Scoring is 5-dimensional: - Functional — do the target tests pass? pass/fail, weighted - Quality — ruff lint + radon complexity + maintainability index - Security — bandit/semgrep scan, high/medium/low findings - Efficiency — steps taken vs task complexity budget - Human-like — LLM judge panel scoring patch naturalness disabled in v0.1 Runs inside a Docker sandbox with network off. Reproducible, isolated, no cheating. First Results: qwythos-9b-q6 on Local GPU We ran 7 tasks today with qwythos-9b-q6 9B parameter quantized model, running via Ollama on a GTX 1080 Ti, 11GB VRAM: | Task | Difficulty | Functional | Total | |------|-----------|-----------|-------| | hello-world | trivial | 1.00 | 0.9948 | | py-ttl-cache-expiry | easy | 0.00 | 0.159 | | py-csv-export-feature | easy | 0.00 | 0.158 | | py-retry-refactor | medium | 0.00 | 0.390 | | py-topo-sort-cycle | medium | 0.00 | 0.158 | | py-jsonpointer | hard | 0.00 | 0.157 | Honest result: 1/6 functional on real tasks. The trivial task was perfect. Everything else hit the step budget 14 steps without producing a passing patch. The interesting one is py-retry-refactor — functional score 0, but quality 0.91 and security 0.95. The model wrote good, clean Python.
Appopoleis