If the slice name or test files are missing, report the error and recommend Revise — there is nothing to review

If acceptance criteria are missing, attempt to infer them from a delivery plan in .stencila/plans/ as a standalone convenience — look for criteria related to the current slice name. In workflow use, the stage prompt should always provide acceptance criteria explicitly.

Parse test files into a list of file paths for reading

Extract: number of tests passed, number failed, failure messages, and any notes about compilation errors or unexpected failures

If no execution results are available, note this as a limitation — the review will assess test quality without execution feedback

Use read_file to load each test file listed in the inputs

If a file does not exist, flag it immediately — this indicates a problem with the test creation step

Use glob to search for test files in the target package directories:

Exclude the test files being reviewed from the convention sample

Use read_file to examine existing tests and learn:

Broaden the search to sibling packages or the project root

If still nothing is found, note this in the review — convention conformance cannot be assessed and only intrinsic quality dimensions apply

First classify the criterion as either:

Determine whether at least one test directly verifies each test-appropriate executable behavior criterion

Check that the test's assertion actually exercises the criterion, not just a superficially related behavior

Flag any test-appropriate executable behavior criteria that have no corresponding test

Do not mark a documentation-only or other non-test-deliverable criterion as missing merely because there is no automated test for it

Flag tests that read source files, comments, or documentation solely to prove prose was added as an inappropriate testing strategy unless the repository already has a strong convention for such checks and the criterion explicitly calls for them

Flag tests that do not map to any acceptance criterion — over-testing adds maintenance burden without providing safety. Assess each unmapped test for triviality using the criteria in Step 7f

File placement: Are test files in the expected location?

Naming: Do test file names and test function names follow the codebase convention?

Framework usage: Are the same test framework and assertion style used?

Import patterns: Do imports follow the existing convention?

Organization: Are tests grouped and structured consistently with existing tests?

Test names should describe the behavior being tested, not the implementation detail

Good: test_parse_returns_malformed_token_error_for_empty_string

Bad: test1, test_parse, test_it_works

Names should be specific enough that a failure message tells you what broke without reading the test body

Each test should have meaningful assertions that verify observable behavior

Assertions should test the right property (return value, side effect, error type — whatever the criterion requires)

Avoid tests that assert only that code runs without error unless that is the specific criterion

Check for over-assertion: testing internal implementation details that may change creates brittle tests

Treat tests that inspect source files or comments only to confirm documentation was written as brittle and generally inappropriate for TDD; normally raise this as at least a Medium-severity quality issue

Each test should be independent — no test should depend on the outcome or side effects of another test

Tests should set up their own state rather than relying on shared mutable state

If fixtures or setup code is shared, it should be clearly defined and not create hidden coupling

Tests should be understandable without reading the implementation

Arrange-Act-Assert or Given-When-Then structure should be discernible

Magic numbers and cryptic test data should be avoided or explained with named constants or comments

The purpose of each test should be clear from its name and body together

Tests should verify what the slice requires, not more

Avoid testing behavior that belongs to other slices or to code that already exists and is already tested

Each test function should focus on one logical behavior

Tautological assertions — the test asserts something that is guaranteed to be true by construction. Examples:

Type-system-verified properties — the test verifies something the compiler or type checker already enforces. Examples:

Constructor echo tests — the test creates an object with known inputs and asserts each field matches the input, without exercising any logic, transformation, validation, or default-value computation. If the constructor has no interesting behavior (no validation, no defaults, no derived fields), testing it is tautological

Getter/setter round-trips with no logic — the test calls a setter and then a getter to confirm the value is stored, when the getter/setter pair is trivial (no validation, no transformation, no side effects)

Existence checks with no behavior — the test imports a symbol or calls a function with no assertions, only to confirm it exists and does not raise an error. This is sometimes called a "smoke test" but provides minimal value when the language's tooling already verifies the symbol exists

String/display representation checks for trivial formats — asserting that a toString(), __str__(), Display, or similar string-conversion output matches a hardcoded string when the implementation is a trivial format string with no conditional logic

Duplicate coverage — the test covers exactly the same behavior and code path as another test in the same suite, with no meaningfully different inputs, edge cases, or assertions

Create false confidence that "we have N tests" when those tests catch nothing

Must be updated during refactors, slowing down legitimate changes

Clutter the test suite, making it harder to find and understand the tests that matter

Waste CI time on every run

Encourage a culture of testing quantity over quality

Check whether the acceptance criteria imply edge cases (empty input, boundary values, null/None, maximum sizes, concurrent access, etc.)

Check whether error paths specified in the criteria have corresponding tests

Flag missing edge-case tests when the criteria clearly require them

Do not penalize the absence of edge-case tests that go beyond the acceptance criteria — test scope should match slice scope

All tests should fail: In the Red phase, tests should fail because the implementation does not exist or is incomplete, not because of test bugs

Correct failure reasons: Check that failure messages indicate missing functions, unresolved imports, unimplemented methods, or assertion failures against stub/missing behavior — not syntax errors, framework misconfiguration, or import errors for existing code

No tests should pass: If some tests pass during Red phase, they may be testing existing behavior rather than new slice behavior — flag this as a concern (it may be acceptable if the slice extends existing code, but it deserves scrutiny)

Compilation/parse errors: If the test runner failed due to compilation or parse errors, this is a significant issue — the tests have syntax or structural problems that must be fixed before they can serve as a specification

All test-appropriate executable acceptance criteria are covered by at least one test (no ❌ Missing in the coverage matrix for criteria that should be tested)

There are no High-severity findings

Red-phase failures (if available) indicate correctly missing implementation, not test bugs

Any test-appropriate executable acceptance criterion is missing test coverage, OR

Any High-severity finding exists (test bugs, syntax errors, serious quality problems, a majority-trivial test suite), OR

Red-phase failures indicate test defects rather than missing implementation

Convention conformance — deviations from codebase test conventions

Test quality — naming, assertions, isolation, readability, minimality issues

Edge cases and error paths — missing coverage for edge cases or error handling

Red-phase validation — problems with how tests fail (or unexpectedly pass)

Indicate severity as High, Medium, or Low

Describe the issue precisely, referencing the specific test function or file

Explain why it matters

High: Missing acceptance-criteria coverage, test bugs, syntax/compilation errors, tests that pass when they should fail, tests that fail for the wrong reason, a test suite where the majority of tests are trivial (more maintenance cost than testing value)

Medium: Convention violations, weak assertions, poor naming, missing edge-case tests that the criteria imply, tests that over-assert on implementation details, tests that verify documentation or comments by reading source files when inspection would be more appropriate, individual trivial tests (tautological assertions, constructor echo tests, type-system-verified properties, getter/setter round-trips with no logic — see Step 7f)

Low: Style inconsistencies, minor naming improvements, opportunities to simplify setup, tests that go slightly beyond slice scope

No test execution results available: Review the tests on their quality dimensions alone. Skip the Red-phase validation step. Note in the report that execution feedback was unavailable. The review can still recommend Accept or Revise based on coverage and quality alone.

Test files do not exist: If the listed test files cannot be found, recommend Revise immediately with a clear finding that the test files are missing.

No existing tests in the codebase for convention discovery: Note that convention conformance cannot be assessed. Evaluate tests on intrinsic quality dimensions only. Do not penalize convention deviations when no conventions could be discovered.

Acceptance criteria are vague: Interpret the criteria as concretely as possible and evaluate coverage against that interpretation. Note which criteria were ambiguous and what interpretation was used. If the vagueness makes it impossible to judge coverage, flag this as a finding but do not automatically recommend Revise — the tests may be reasonable given the available information.

Acceptance criteria include documentation or prose requirements: Mark those rows in the coverage matrix as ➖ Non-test deliverable unless the project already has an established automated check for them and the criterion clearly intends that. Do not recommend Revise solely because there is no unit test asserting docs exist.

Tests intentionally pass during Red phase: Some tests in a slice that extends existing code may pass because the existing code already satisfies part of the criterion. If the slice scope explains this, note it as acceptable. If it is unexpected, flag it for scrutiny.

Very large test files: For test files with many tests, focus the review on the tests relevant to the current slice's acceptance criteria. Existing tests in the file that predate the current slice are outside the review scope.

Multiple test files across packages: Review each file against its own package's conventions. Produce a single unified report covering all files.

Test creation agent added tests beyond acceptance criteria: Extra tests that verify meaningful behavior beyond the acceptance criteria are acceptable — flag as Low severity only if they risk confusion. However, if the extra tests are trivial (see Step 7f), flag them at Medium or High severity as appropriate. A test that goes beyond the acceptance criteria and is trivial should be removed, not merely noted.

AI-generated test suites with inflated counts: AI test generators commonly produce suites padded with trivial tests — constructor echo tests, type-verification tests, and tautological assertions — to give the appearance of thorough coverage. Be especially vigilant with AI-generated tests: scrutinize every test against the triviality criteria in Step 7f and do not accept a suite at face value just because it has many tests.

Tests were added only to prove docs exist: Recommend Revise unless there is an explicit repository convention or plan requirement for an automated docs-presence check. Prefer feedback telling the creator to remove those tests and leave the documentation requirement to implementation and human review.