Accessibility Test Cases

A11Y-QC-002

Version-frozen calibration set

Generated file current Authoring: Content complete
What file status means

This status reports only whether the generated procedure matches the current controlled source hash. It is not independent content verification, regulatory approval, product evidence or a conformance decision.

Recorded standards review: No optional specialist review recorded. Next review: Not scheduled.

MethodQuality-control review
TesterAccessibility quality lead
Indicative effort125–181 min small scope

Have assigned reviewers independently score version-frozen examples, measure agreement and resolve differences without rewriting their original decisions.

Available formats

Use this test in the format you need

The HTML procedure includes the working worksheet. JSON provides the structured source data. Editable Markdown files are delivered inside package downloads.

Purpose

What this test checks

Have assigned reviewers independently score version-frozen examples, measure agreement and resolve differences without rewriting their original decisions.

What this result means: This procedure determines whether Run the version-frozen execution-team calibration meets its stated expectations for the recorded targets and accessibility-quality-control. Unexecuted targets, states and environments remain outside that result.

Why it matters

Detect inconsistent procedure interpretation and unreliable decisions before production findings support claims. People most likely affected: people affected by assessment decisions whose reliability depends on consistent, evidence-based procedure execution. Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. In the recorded accessibility-quality-control environment, use frozen fixtures, independent scoring and documented reconciliation; unresolved assessor disagreement can turn the same evidence into inconsistent client findings.

When to use it

Apply before execution agents use a new or materially revised procedure for production assessment and at the governance-defined recalibration interval.

Confirm before starting

Mark it not applicable only when

Exceptions

This project quality-control method has no normative exception. Any permitted exclusion must be defined in the frozen quality plan and reported as a coverage limitation.

Prepare

What you need

  • The product area and test targets are agreed before testing starts.
  • The operating system, browser, viewport and input method are recorded.
  • Test data is safe to use and does not expose unnecessary personal information.

Targets and required variations

  • Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
  • Passed, failed, inapplicable, cantTell and untested outcomes.
  • Material exceptions, borderline cases and at least one evidence-quality failure where meaningful.

Clean state

  • Complete a known-good calibration in the same environment before product testing.
  • Start from a recorded clean or known state and preserve any prior preferences, extensions, cache or profile settings that can affect output.

Effort estimate

Setup
30 min
Per target
45 min
Evidence
20 min
Technical review
30 min

Assumptions: one frozen review set and one reviewer pair; disagreements, adjudication and reruns are additional effort

Variation: Add a new per-target execution and evidence allowance for each additional target class or state that could produce a different result. Add setup time for each distinct environment.

Excluded: Client coordination, inaccessible-environment recovery, recruitment, legal review, remediation implementation and waiting time unless the procedure states otherwise.

Execute

Test steps

Follow each action in order. The expected result is always visible; open the guidance when you need setup, tool, evidence, decision or repeat details. Step-level recording is optional while work is in progress and required only to export a completed execution.

Step 1

Freeze the calibration package

Purpose: For Version-frozen calibration set, prepare the named scenario: Freeze the calibration package. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. The preparation must establish this result: The package and decision thresholds cannot be changed after results are known without a new versioned run.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Freeze the calibration package.

Record the catalogue, procedure and fixture versions; expected outcome key; exception and borderline coverage; assigned execution-agent IDs and versions; distinct verification-agent ID and protocol; conflicts; agreement metric; threshold; and rerun rule before revealing any answers.

Expected result

The package and decision thresholds cannot be changed after results are known without a new versioned run.

Failure observation

During Version-frozen calibration set, the target, starting state, environment or scope record remains ambiguous for the scenario named Freeze the calibration package. The preparation result cannot be demonstrated: The package and decision thresholds cannot be changed after results are known without a new versioned run.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
Immutable calibration package before execution-agent assignment.
Tool or command
Version control, checksums and a blinded answer-key store.
Record
Versions/checksums, coverage matrix, agent IDs and versions, conflicts, metric formula, threshold and answer-key integrity record.
Pass
Record passed for the Version-frozen calibration set scenario named Freeze the calibration package only when the evidence (Versions/checksums, coverage matrix, agent IDs and versions, conflicts, metric formula, threshold and answer-key integrity record.) supports: The package and decision thresholds cannot be changed after results are known without a new versioned run.
Fail
Record failed when the action is completed and this failure observation is reproduced: During Version-frozen calibration set, the target, starting state, environment or scope record remains ambiguous for the scenario named Freeze the calibration package. The preparation result cannot be demonstrated: The package and decision thresholds cannot be changed after results are known without a new versioned run.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Create a successor version rather than altering the active package.
Limitations
This scenario records the action named Freeze the calibration package for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
The expected answer key must pass a separate hash-bound agent verification; unresolved normative interpretation remains cantTell.
Record this step

Step 2

Collect blinded independent decisions

Purpose: For Version-frozen calibration set, execute the named scenario: Collect blinded independent decisions. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. Compare the observation with this result: Original decisions are complete, independently produced and immutable.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Collect blinded independent decisions.

Have every execution agent execute every assigned fixture independently using the frozen procedure. Record target outcome, evidence, rationale, exception use and time only where the plan requires it. Do not expose expected or peer answers.

Expected result

Original decisions are complete, independently produced and immutable.

Failure observation

After the tester completes the Version-frozen calibration set scenario named Collect blinded independent decisions, the recorded observation for the target is inconsistent with the required result: Original decisions are complete, independently produced and immutable.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
Identical fixture and procedure versions for every execution agent.
Tool or command
Normal execution worksheet with blinded answer access.
Record
Agent/fixture IDs, versions, outcomes, evidence, rationale, exception decisions and completion timestamp.
Pass
Record passed for the Version-frozen calibration set scenario named Collect blinded independent decisions only when the evidence (Agent/fixture IDs, versions, outcomes, evidence, rationale, exception decisions and completion timestamp.) supports: Original decisions are complete, independently produced and immutable.
Fail
Record failed when the action is completed and this failure observation is reproduced: After the tester completes the Version-frozen calibration set scenario named Collect blinded independent decisions, the recorded observation for the target is inconsistent with the required result: Original decisions are complete, independently produced and immutable.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Preserve originals and schedule a new blinded rerun after corrective action.
Limitations
This scenario records the action named Collect blinded independent decisions for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
Never replace the first decision with the corrected answer; calibration measures the original independent judgement.
Record this step

Step 3

Calculate agreement and error patterns

Purpose: For Version-frozen calibration set, execute the named scenario: Calculate agreement and error patterns. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. Compare the observation with this result: The metric, threshold comparison and material disagreement patterns are reproducible.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Calculate agreement and error patterns.

Compare independent decisions with the separately reviewed answer key and with each other using the predeclared metric. Report confusion among all five outcomes, exception disagreements and evidence-quality failures.

Expected result

The metric, threshold comparison and material disagreement patterns are reproducible.

Failure observation

After the tester completes the Version-frozen calibration set scenario named Calculate agreement and error patterns, the recorded observation for the target is inconsistent with the required result: The metric, threshold comparison and material disagreement patterns are reproducible.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
All original records locked.
Tool or command
Reproducible calculation or analysis script with saved inputs and output.
Record
Input record hashes, calculation version, metric result, confusion matrix, disagreements and threshold decision.
Pass
Record passed for the Version-frozen calibration set scenario named Calculate agreement and error patterns only when the evidence (Input record hashes, calculation version, metric result, confusion matrix, disagreements and threshold decision.) supports: The metric, threshold comparison and material disagreement patterns are reproducible.
Fail
Record failed when the action is completed and this failure observation is reproduced: After the tester completes the Version-frozen calibration set scenario named Calculate agreement and error patterns, the recorded observation for the target is inconsistent with the required result: The metric, threshold comparison and material disagreement patterns are reproducible.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Correct calculation defects transparently with a versioned report; do not alter source decisions.
Limitations
This scenario records the action named Calculate agreement and error patterns for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
A single headline percentage can hide systematic confusion between failed, cantTell and untested outcomes.
Record this step

Step 4

Verify causes and corrective actions

Purpose: For Version-frozen calibration set, execute the named scenario: Verify causes and corrective actions. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. Compare the observation with this result: Every material disagreement has an evidence-supported resolution or remains an open accuracy dispute.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Verify causes and corrective actions.

Have the distinct verification agent inspect each material disagreement against the frozen method and evidence. Classify method ambiguity, source dispute, capability gap, fixture defect or evidence failure and assign corrective action without inventing consensus; use cantTell when the evidence does not support a cause.

Expected result

Every material disagreement has an evidence-supported resolution or remains an open accuracy dispute.

Failure observation

After the tester completes the Version-frozen calibration set scenario named Verify causes and corrective actions, the recorded observation for the target is inconsistent with the required result: Every material disagreement has an evidence-supported resolution or remains an open accuracy dispute.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
Original decisions and calculation locked and available to the verification agent.
Tool or command
Hash-bound agent-verification record.
Record
Disagreement, evidence, classification, decision, verifier ID/protocol, owner, due date and affected procedure/fixture version.
Pass
Record passed for the Version-frozen calibration set scenario named Verify causes and corrective actions only when the evidence (Disagreement, evidence, classification, decision, verifier ID/protocol, owner, due date and affected procedure/fixture version.) supports: Every material disagreement has an evidence-supported resolution or remains an open accuracy dispute.
Fail
Record failed when the action is completed and this failure observation is reproduced: After the tester completes the Version-frozen calibration set scenario named Verify causes and corrective actions, the recorded observation for the target is inconsistent with the required result: Every material disagreement has an evidence-supported resolution or remains an open accuracy dispute.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Version procedure or fixtures when corrective action changes their meaning, then rerun.
Limitations
This scenario records the action named Verify causes and corrective actions for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
Post-scoring agreement is not the original inter-rater agreement and must remain separate.
Record this step

Step 5

Verify corrective action with a rerun

Purpose: For Version-frozen calibration set, execute the named scenario: Verify corrective action with a rerun. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. Compare the observation with this result: The cohort meets the threshold without an unresolved material dispute.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Verify corrective action with a rerun.

After versioned capability, method or fixture changes, run a new blinded calibration set that includes equivalent but unrevealed examples. Recalculate the same predeclared metric.

Expected result

The cohort meets the threshold without an unresolved material dispute.

Failure observation

After the tester completes the Version-frozen calibration set scenario named Verify corrective action with a rerun, the recorded observation for the target is inconsistent with the required result: The cohort meets the threshold without an unresolved material dispute.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
Corrective action completed; new frozen package and hidden answers.
Tool or command
Same controlled calibration workflow.
Record
New package/version, capability or method change, independent decisions, metric and comparison with the prior run.
Pass
Record passed for the Version-frozen calibration set scenario named Verify corrective action with a rerun only when the evidence (New package/version, capability or method change, independent decisions, metric and comparison with the prior run.) supports: The cohort meets the threshold without an unresolved material dispute.
Fail
Record failed when the action is completed and this failure observation is reproduced: After the tester completes the Version-frozen calibration set scenario named Verify corrective action with a rerun, the recorded observation for the target is inconsistent with the required result: The cohort meets the threshold without an unresolved material dispute.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Repeat only under a new versioned run with a documented reason.
Limitations
This scenario records the action named Verify corrective action with a rerun for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
Reusing revealed examples tests memorisation, not reliable application of the method.
Record this step

Step 6

Repeat material variations and recovery

Purpose: For Version-frozen calibration set, execute the named scenario: Repeat material variations and recovery. Target: Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture. Compare the observation with this result: Every material variation and required recovery path has its own reproducible observation; no unexecuted state is merged into a passing result.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Repeat material variations and recovery.

Repeat the Version-frozen calibration set action for each inventoried variation that can change the outcome. Include relevant default, revealed, error, interruption, recovery and confirmation states and complete the selected process from start to finish where this test can affect task completion.

Expected result

Every material variation and required recovery path has its own reproducible observation; no unexecuted state is merged into a passing result.

Failure observation

After the tester completes the Version-frozen calibration set scenario named Repeat material variations and recovery, the recorded observation for the target is inconsistent with the required result: Every material variation and required recovery path has its own reproducible observation; no unexecuted state is merged into a passing result.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
For Version-frozen calibration set: Passed, failed, inapplicable, cantTell and untested outcomes. Material exceptions, borderline cases and at least one evidence-quality failure where meaningful.
Tool or command
For Version-frozen calibration set, use the same recorded method, then change only the state, environment or recovery condition being compared.
Record
Variation identifier; changed state or environment; start and end points; actual observation; recovery result; intervention; evidence reference and target outcome.
Pass
Record passed for the Version-frozen calibration set scenario named Repeat material variations and recovery only when the evidence (Variation identifier; changed state or environment; start and end points; actual observation; recovery result; intervention; evidence reference and target outcome.) supports: Every material variation and required recovery path has its own reproducible observation; no unexecuted state is merged into a passing result.
Fail
Record failed when the action is completed and this failure observation is reproduced: After the tester completes the Version-frozen calibration set scenario named Repeat material variations and recovery, the recorded observation for the target is inconsistent with the required result: Every material variation and required recovery path has its own reproducible observation; no unexecuted state is merged into a passing result.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Repeat this action for every required variation of Run the version-frozen execution-team calibration and after a material target, content, state, environment or configuration change. Preserve the original observation.
Limitations
This scenario records the action named Repeat material variations and recovery for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
For Version-frozen calibration set, change one material condition at a time where practical. Do not assume that a passing default state covers errors, overlays, orientation changes, interruptions or recovery.
Record this step

Step 7

Record the scoped quality decision

Purpose: For Version-frozen calibration set, complete the record for the scenario named Record the scoped quality decision and preserve evidence that demonstrates this result: The decision is scoped to identified reviewers, procedure version, suite version and time period.

User impact: Uncalibrated reviewers may reach conflicting results, hide uncertainty or create misleading accessibility claims. This scenario isolates that risk in the action named Record the scoped quality decision.

A distinct competent reviewer records pass, fail, cantTell or untested for this cohort and procedure version, the production-use limitation and next calibration date. Preserve all source records and keep any organisational release decision separate.

Expected result

The decision is scoped to identified reviewers, procedure version, suite version and time period.

Failure observation

The Version-frozen calibration set evidence or decision record for the scenario named Record the scoped quality decision does not demonstrate the required result: The decision is scoped to identified reviewers, procedure version, suite version and time period.

Setup, evidence and decision guidance
Target
Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
Required state
All required review findings and reruns complete.
Tool or command
Quality-control register and immutable evidence references.
Record
Reviewer identity, method/version, decision time, scope, limitations, source-record references, next review and status effect.
Pass
Record passed for the Version-frozen calibration set scenario named Record the scoped quality decision only when the evidence (Reviewer identity, method/version, decision time, scope, limitations, source-record references, next review and status effect.) supports: The decision is scoped to identified reviewers, procedure version, suite version and time period.
Fail
Record failed when the action is completed and this failure observation is reproduced: The Version-frozen calibration set evidence or decision record for the scenario named Record the scoped quality decision does not demonstrate the required result: The decision is scoped to identified reviewers, procedure version, suite version and time period.
Not applicable
No execution-agent cohort is assigned to use the procedure.
Blocked
Record blocked, not passed, when The target is applicable, but a required state, environment, action or evidence item was not completed.
Repeat or recover
Recalibrate after material source, procedure, platform, adapter or reviewer change.
Limitations
This scenario records the action named Record the scoped quality decision for Version-frozen calibration set in accessibility-quality-control. It does not establish behaviour for another target, state or environment and does not set a WCAG conformance decision.
If you are new to this test
The reviewer cannot create legal clearance, participant evidence, conformance or organisational approval.
Record this step

Decide

How to decide

Passed

Every assigned execution agent completes the frozen suite independently, the predeclared agreement threshold is met, exception and evidence decisions are defensible, no material accuracy dispute remains and the distinct verification agent records the scoped decision.

Failed

The cohort misses the predeclared threshold, a material systematic error remains, independence is compromised or corrective action fails verification.

Not applicable
No execution-agent cohort is assigned to use the procedure.
Cannot tell
The answer key, metric inputs, independence or cause of material disagreement cannot be validated.
Not tested
The target is applicable, but a required state, environment, action or evidence item was not completed.
Aggregation and reporting boundaries

Aggregation: Preserve outcomes per execution agent and fixture. Apply only the predeclared metric and threshold; cantTell and untested prevent a passed control.

WCAG decision: This control makes no WCAG decision. It supports method reliability and execution-agent readiness within its recorded scope.

Automation: Deterministic calculation and agent verification may decide the recorded quality-control scope but cannot create legal clearance, participant evidence, WCAG conformance or organisational approval.

Record

Record the result

Record the target, state and observed behaviour. Add evidence or technical details only when needed.

Choose the option that best matches what happened to this target.

Describe the actual behaviour and why it supports the selected result.

When more than one target is recorded, the overall procedure result is calculated from all target results.

Environment: Not set.
Add evidence or details

Choose only the group needed for this record. These fields remain optional for a working draft.

Evidence references

Enter one file name, issue ID or evidence reference per line. Files stay on your device.

Test environment

Save a named profile once to reuse the same product, platform and environment details on other procedures in this browser.

Engagement and tester identifiers

Component expectation results

Use these controls only when the procedure has distinct component expectations. The overall outcome must agree with them.

E1 Procedure, fixture, answer key, cohort, metric and threshold are frozen before scoring.

E2 Original independent decisions and evidence are preserved and produce a reproducible agreement result.

E3 The threshold is met with no unresolved material accuracy dispute and a distinct agent-verification decision is recorded.

Formal completion and limitations

Identify the frozen list or sampling record used to select every target represented by this execution.

Choose a final result before marking the execution complete. Completed records require the full environment, step, evidence and method-specific information.

Data management

Export before clearing if the record must be retained. Apply the engagement retention schedule and delete browser data when it is no longer needed.

Saved drafts stay in this browser and are not uploaded. Do not enter credentials, secrets or unnecessary personal data.

Reference

Supporting information

Standards and requirement mappings
  • W3C-WCAG-EM-1 · W3C-WCAG-EM-1 advisory · supporting

    This source does not override WCAG or create an additional WCAG conformance requirement.

  • W3C-WCAG-EM-2 · W3C-WCAG-EM-2 advisory · supporting

    This source does not override WCAG or create an additional WCAG conformance requirement.

  • W3C-ACT-RULES-FORMAT · W3C-ACT-RULES-FORMAT advisory · supporting

    This source does not override WCAG or create an additional WCAG conformance requirement.

Supporting and advisory guidance

Supporting sources are identified in the mappings and source register. They do not add conformance requirements.

Evidence specification
  • Frozen package

    Catalogue/procedure/fixture versions and hashes, reviewed answer-key custody, cohort, conflicts, metric and threshold.

    Use a redacted extract or evidence reference. Do not include credentials, secrets or unnecessary personal data.

  • Independent source records

    Every auditor's original outcomes, evidence, rationale and timestamps.

    Use a redacted extract or evidence reference. Do not include credentials, secrets or unnecessary personal data.

  • Agreement calculation

    Inputs, calculation version, metric, confusion matrix and threshold comparison.

    Use a redacted extract or evidence reference. Do not include credentials, secrets or unnecessary personal data.

  • Adjudication and action

    Disagreement classification, evidence, decision, owner, changes and rerun result.

    Use a redacted extract or evidence reference. Do not include credentials, secrets or unnecessary personal data.

  • Quality review

    Distinct reviewer identity, method/version, decision time, scope, limitations and next calibration date.

    Use a redacted extract or evidence reference. Do not include credentials, secrets or unnecessary personal data.

Worked example and outcome calibration

A11Y-QC-002-WORKED-001: representative worked example

Context: Representative synthetic example for Version-frozen calibration set on a fictional, version-frozen product target; no client or participant execution occurred.

Inspected: The tester follows S2 (Collect blinded independent decisions) using the target, required state, tool and repeat rule stated in this procedure.

Observed output: After the tester completes the Version-frozen calibration set scenario named Collect blinded independent decisions, the recorded observation for the target is inconsistent with the required result: Original decisions are complete, independently produced and immutable.

Evidence:

  • example-evidence-001: target and state record for Each execution agent's independent outcome and evidence for every passed, failed, inapplicable, cantTell and untested calibration fixture.
  • example-evidence-002: contemporaneous observation from the named method — Agent/fixture IDs, versions, outcomes, evidence, rationale, exception decisions and completion timestamp.
  • example-evidence-003: comparison note linking the actual observation to the expected result and claim boundary

Tester reasoning: The recorded observation does not demonstrate the expected result: Original decisions are complete, independently produced and immutable. The failed decision is limited to this synthetic target and state; unexecuted coverage is not inferred.

Decision: failed

Finding: A11Y-QC-002-example-finding-001 — Run the version-frozen execution-team calibration: expected result is not demonstrated. The represented barrier may affect People affected by assessment decisions whose reliability depends on consistent, evidence-based procedure execution.

Remediation: Change the represented target or process so it demonstrates this result: Original decisions are complete, independently produced and immutable. Then rerun the affected step and every dependent variation.

Reviewer feedback: The reviewer confirms applicability, environment, actual-versus-expected comparison, evidence references and bounded wording; unsupported product-wide or conformance language is removed.

Final client wording: Synthetic training finding: the sampled Run the version-frozen execution-team calibration target does not meet the procedure expectation in the recorded state. User impact and broader scope require evidence from the buyer's real execution.

Retest: Synthetic retest example: after the represented change, repeat the failed step plus material error and recovery states. Close the finding only when new evidence demonstrates the expected result; otherwise retain the non-pass outcome.

Boundary: This is a representative, synthetic worked example for training and calibration. It is not evidence from a client product, participant or real assistive-technology execution.

Outcome calibration

Passed
The blinded execution-agent cohort meets the predeclared threshold on a frozen suite, no material dispute remains and a distinct verification agent records the scoped readiness decision.

All quality-control gates are met for the identified cohort and version.

Failed
Auditors systematically mark blocked executions as cantTell target outcomes and the verified rerun still misses the threshold.

The control fails because outcome architecture remains unreliable.

Not applicable
No auditor is assigned to use this procedure in production.

The control has no cohort target; the procedure is not thereby approved.

Cannot tell
The answer key's independent review record is missing, so agreement with it cannot establish correctness.

The evidence cannot support a defensible quality decision.

Not tested
The suite is frozen, but one assigned auditor has not completed the cases.

The cohort calibration remains incomplete.

Sampling, complete processes and platform limitations

Apply this control to the preselected assessment sample. Do not change the sample or threshold after results are known.

If this check is part of a user journey, test the selected journey from start to finish, including relevant errors and recovery. Record any path that was not tested.

  • Quality-control evidence measures method reliability; it is not product accessibility or conformance evidence.
  • Check relevant browser, tool or assistive-technology release notes before assigning unexpected behaviour to the product.
  • If responsibility is unclear, record the reproducible behaviour and use cannot tell until it is resolved.
Related procedures

No related procedure is recorded.

ACT rules and WCAG technique traceability

This is a project-authored manual procedure, not a W3C Approved ACT Rule. An ACT result can support only the part of this procedure covered by that rule. It does not replace the remaining manual or assistive-technology checks.

No supporting W3C Approved ACT Rule is selected for this procedure.

This specialist procedure is not assigned to one WCAG success criterion.

Review history and source register
  1. 2026.08.27-rc.9 — Reworked for plain language, case-focused steps, scoped evidence and transparent draft review status.
  2. 2026.08.27-rc.9 — Migrated to professional catalogue schema v3 with separated outcomes, structured authority, applicability, evidence, decision and review records.
  3. 2026.08.27-rc.9 — Specialist procedure mapped to its task family and current first-party platform guidance and checked for internal consistency; competent review and real execution remain separate.
  4. 2026.08.27-rc.9 — Replaced migrated boilerplate with a case-specific reference implementation and completed generated-file integrity review.
  5. 2026.08.27-rc.9 — Added explicit preparation and variation/recovery steps plus a bounded representative worked example for paid-template use.
  6. 2026.08.27-rc.9 — Procedure guidance and presentation simplified without changing draft governance status.

Review decision: pending. Open issues: No production calibration run, live execution evidence, agreement metric or organisational release decision is claimed.; The calibration record is an internal quality control, not independent verification; production release authority remains separate.

Source historyReport an accuracy issue
WCAG
Specialist or advisory method
Execution
Manual test
Catalogue version
2026.08.27-rc.9