Evaluation that can say not yet.

Forge separates evaluation from generation, blinds system identity, measures reviewer agreement, reverses judge order, and keeps model-quality claims locked until the declared evidence exists.

Current evidence: the engineering controls pass on an eight-item authored fixture. Genuine human annotations have not been collected, so human calibration and model quality remain unestablished.

What is frozen

Evaluation release

Items, protocol, packets, review sheet, and blinding key are fingerprinted under one deterministic evaluation ID.

Human packet

Reviewers see Candidate A and Candidate B without generator identity or position mappings.

Judge packet

Every comparison appears in primary and reversed order so position consistency is measured directly.

Controlled fixture result

0.754 Fixture Krippendorff alpha

0.875 Fixture judge agreement

0.875 Position consistency

Two fixture annotators produced 16 labels over eight items. The report also records Cohen kappa 0.741935, judge macro F1 0.858586, five Forge wins, one baseline win, and two ties. The paired fixture win rate is 0.75 with a 95 percent bootstrap interval from 0.4375 to 1.0. This small controlled set tests the machinery and failure semantics. It is not a performance benchmark.

The claim gate

Requirement	Declared boundary	Current status
Independent evaluation set	Frozen outside generator context	Framework ready, real set pending
Human coverage	At least 200 genuine human-reviewed items	Pending
Overlap	At least 50 multiply reviewed items	Pending
Reviewer agreement	Nominal alpha at least 0.667	Pending genuine labels
Judge independence	Different family from the candidate generator	Enforced in code
Judge alignment	At least 0.7 exact agreement with human consensus	Pending genuine labels
Order robustness	At least 0.8 primary versus reversed consistency	Enforced in code

Reviewer workflow

Freeze independently authored comparisons and the annotation protocol.
Give reviewers only the generated human_review_sheet.csv.
Collect A, B, tie, or both-bad labels with confidence and reason codes.
Lock the labels before exposing the blinding key.
Run a versioned judge from a different model family over both orders.
Publish agreement, slice results, bias checks, intervals, limitations, and failure cases.

make evaluation-fixture

python scripts/prepare_evaluation.py --help
python scripts/run_pairwise_judge.py --help
python scripts/analyse_evaluation.py --help

Why the report can still fail

A controlled fixture cannot become human evidence by changing a label. A same-family judge cannot support a headline comparison. Incomplete review coverage, weak reviewer agreement, low judge agreement, or sensitivity to response order keeps the report in controlled_or_incomplete status. Judge alignment also does not prove that training improved a downstream model.

Read the complete annotation protocol, inspect the technical controls, or run the browser demo.