We are auditing the checker and deleting what doesn't hold up

Correction: Basilisk has retracted its former perfect-score claim. The result was not a trustworthy measure of specification conformance. We asked for Basilisk to be removed from the python/typing results table, and it has been removed. Basilisk's current conformance percentage is temporarily unknown, and we are not trying to restore it.

We found checker logic fitted to the exact contents of conformance test files rather than implementing the typing specification generally. Those rules matched the spelling of code rather than its meaning: type-alias validation used prefixes and substrings taken from raw source text, including a special case for eval( purely because that spelling appeared in one test file. Rename an import or reformat a file and the answer changed, even though the typing behavior being tested had not.

A passing result from code developed against the exact fixtures is not evidence, so the fix is not a better score.

What we are doing

We are auditing every rule and deleting the ones that don't do real type checking. Not rewriting them, not patching them, not marking them TODO — deleting them, and leaving a failing test behind so the gap is visible rather than hidden. A rule stays only if it decides from the resolved syntax tree and returns the same diagnostics when the same program is spelled differently.

The consequences are deliberate, and we would rather state them up front than have you discover them:

  • Basilisk gets smaller before it gets better. Expect fewer rules and fewer diagnostics.
  • The conformance number will fall. That is the correct outcome of removing logic that was never doing the analysis, and we will report each drop rather than avoid it.
  • A failing test is worth more to us than a passing fixture that was carried by code which doesn't analyse anything. The first is an accurate record of what Basilisk cannot do; the second is a claim that it can.

What is left will be code that is honest about what it does — nothing else.

Whether the deleted analysis gets rebuilt from the specification, or the extension ends up driven by an established open-source checker, is a decision we have not made yet. Either way, no replacement percentage gets published until it survives the robustness testing described below.

Current python/typing results ↗ Full integrity audit ↗ Original bug report ↗ Integrity remediation tracker ↗ Python typing spec ↗ Conformance README ↗

Scope of the audit

The review covers every place a narrow fixture could have stood in for a general implementation: source-text predicates and substring matching, hard-coded symbol spellings, rules organised around a test file rather than a specification concept, duplicated logic, and accept-everything fallbacks standing in for checks that were never written.

Each finding is handled the same way — a test that fails because of the code, then the code is removed, then the removal is recorded. Nothing is quietly repaired in place, because a repair preserves the claim that the rule worked.

This is active remediation, not an indefinite withdrawal. If a defensible result is lower than the former claim, we publish the lower result.

The new publication bar

A future conformance result must satisfy all of these checks:

  1. Run the official, unmodified python/typing harness against Basilisk's default configuration.
  2. Apply AST-preserving mutations such as consistent renaming of type variables and equivalent spelling changes. A rule is not accepted if those changes move its result.
  3. Pass independent off-suite cases derived from the typing specification and real-world code rather than from the upstream fixture text.
  4. Add regression and mutation tests for every test-specific implementation found by the audit.
  5. Publish the robustness and off-suite results alongside the suite percentage and make the methodology reproducible.
  6. Pass an audit by someone outside this project before Basilisk is submitted to python/typing again.

Until that work is complete, old conformance tables, charts, category scores, pass counts, and false-positive totals are withdrawn and should not be cited as Basilisk's current state. We are not quoting a current figure either — a number is not what is wrong here, and publishing a new one before the audit finishes would repeat the mistake.

The same review failure means our published benchmark figures also require revalidation. They are retained only as a clearly labelled historical record on the benchmarks page and must not be used to compare Basilisk with other tools. New performance figures will be published only after the methodology and results have passed the integrity review.