A small offline benchmark that refuses to reward refusing

A small Python package, released on 5 November 2026: enquiry-bench scores routing, cited draft replies and abstention on a fictional service-enquiry task. It runs offline, has no dependencies, and needs no model account.

Assistants that answer service enquiries fail in three separate ways: routing to the wrong place, answering when they should ask or decline, and citing sources that do not support the reply. A single accuracy number hides these, and can reward a system that simply declines everything.

enquiry-bench scores all three for a fictional bike workshop. You give it a prediction file; it checks the route, the abstention flag, an exact reply from an authored list, and the citations, and it writes JSON, CSV, a Markdown report and an SVG confusion matrix. It has no dependencies, makes no network calls and needs no model account.

The baseline problem

The bundled dataset has 48 synthetic enquiries authored for the project. The workshop, its prices and every message are invented; no real business or customer data is used. Half the enquiries should be answered and half should get a clarifying question or a refusal. A baseline that refuses everything gets:

route accuracy 12/48 and joint task pass 12/48
abstention recall 24/24, but abstention precision only 24/48
answered coverage 0/48

An authored reference file that matches the gold scores 48/48 on joint task pass. It is a check that the scorer works, not evidence that any system can reach that score. Both prediction files are synthetic; neither is output from a model.

What the score means, and what it does not

The headline metric needs everything right at once: route, flag, an exact acceptable reply and the exact citations. Every ratio shows its numerator and denominator, and an empty denominator is unavailable, not 0 or 1. A malformed prediction file is rejected; a well-formed wrong answer counts as wrong.

Reply matching is exact string comparison, so a correct answer in other words fails: a deliberate, limited rubric, not a judge of meaning. A citation that exists is not proof the reply is true. The data is small and synthetic, so it says nothing about real-world model performance. The figures above are designed fixture results, not measurements of any model.

What has been checked

On Windows, with CPython 3.12.12, 3.13.12 and 3.14.5: the 36-test suite passed in full on each, with no failures, errors or skips. Its link tests create real symlinks, a hard link and a junction, and in those runs every link was created, so each check that a linked path is rejected actually ran. The 2026-11-03 run was non-elevated, with no administrator rights, Developer Mode or permission changes. In an earlier session, creating those links was refused (access denied), even by a standalone check outside the package, so two link checks could not run; the cause was never established. The tests now skip such a recognized refusal with its reason, and a skip is not a pass.

The 0.1.0 wheel, installed without dependencies into a fresh environment outside the source tree, reproduced the demo figures; its ten runtime files are byte-identical to those of the wheel validated two days before release. An anonymous clone of the public repository passed the same 36 tests, and the package installed from PyPI reproduced the demo. Other platforms are unverified. The tests are not a security guarantee, and path checks are subject to time-of-check races.

Try it

pip install enquiry-bench
enquiry-bench demo --output demo-out

The package is on PyPI as enquiry-bench. From a clone of the repository, python -m pip install --no-deps . and the same demo command also work. Version 0.1.0 is tagged v0.1.0; the wheel and source distribution are attached to the release with their SHA-256 sums.