Deterministic provenance

Document identifiers are SHA-256 digests of normalised source content. TextChunk identifiers combine the document ID, chunk index, and normalised chunk text. Moving a file does not change its source identity, while changing its content does.

document_id = sha256(normalise(document_content))
chunk_id    = sha256(document_id, chunk_index, normalise(chunk_text))

Content identity is intentional. If identical content appears under two file paths, Forge treats it as one source group so the duplicate cannot cross train and test.

Source-grouped split

The earlier splitter shuffled examples inside task groups. That allowed examples from one document, or even one chunk, to appear in both partitions. The rebuilt splitter:

1. Requires document_id on every row.
2. Requires at least two unique sources.
3. Assigns complete source groups to one partition.
4. Searches deterministic seeded candidates for the whole-source subset closest to the requested row fraction.
5. Verifies document and available chunk identifiers are disjoint.
6. Writes an audit manifest with counts, hashes, seed, and overlap results.

Source and record governance

Source governance runs before any generation call. Candidate builds require a policy record for every loaded file with origin, license, rights holder, and permitted uses. Unknown rights are allowed only in smoke mode. A source without permitted training use is quarantined and prevents the candidate from passing.

Deterministic privacy checks cover structured identifiers including email addresses, payment cards with a Luhn checksum, Australian tax file numbers with a checksum, US Social Security number patterns, IPv4 addresses, and Australian phone patterns. The configured action is reject or redact. Generated records then cross a second schema and privacy boundary.

Multi-layer deduplication

1. Normalised exact text groups capture punctuation and casing variants.
2. Three-token word shingles produce 64-value deterministic MinHash signatures.
3. Thirty-two LSH bands propose fuzzy candidate pairs.
4. Exact Jaccard similarity verifies candidates against the configured threshold.
5. Candidate builds add embedding cosine similarity with a recorded model and revision.

Duplicate components are resolved deterministically. The highest quality score wins, with record identity as the tie-breaker. Rejected rows keep duplicate_of, detector reason codes, and similarity evidence in dedupe_rejections.jsonl.

Pipeline identity and resume

The installed forge command, repository wrapper, tests, and future workers share forge.workflow.run_forge. Each stage declares typed configuration, content inputs, expected outputs, model identity, prompt identity, and an implementation version.

A cache key is derived from that contract and the current input fingerprints. A cache hit is accepted only after every recorded output is hashed again. Missing or modified output files invalidate the stage. A failed run records the error and resumes from the last verified stage.

forge \
  --source ./sample_docs \
  --output-dir runs/example \
  --benchmark-file sample_benchmarks/contamination_smoke_test.jsonl \
  --dry-run

pipeline_events.jsonl records start, completion, failure, and cache-hit events. Model and prompt identities are stored as metadata and hashes. API keys are never event fields.

Layered contamination gate

The contamination stage indexes benchmark text for lexical n-gram overlap, compares word shingle Jaccard similarity, and optionally compares embedding cosine similarity. Candidate configuration refuses a disabled semantic backend. A missing benchmark, empty index, or flagged record stops the run after writing the evidence report.

python scripts/check_contamination.py \
  --input run/judged.jsonl \
  --benchmark /path/to/independent_eval.jsonl \
  --output run/contamination_report.json \
  --fail-on-contamination

Embeddings improve paraphrase sensitivity but do not make contamination detection complete. Related yet distinct records can be false positives, and model-specific thresholds can drift. Forge records the model, revision, thresholds, closest benchmark record, and reason codes.

Controlled calibration

make curation-calibration evaluates exact and fuzzy controls on 12 labelled pairs. The fixture includes exact variants, near duplicates, semantic paraphrases, unrelated pairs, and hard negatives. It reports precision, recall, F1, and a 95 percent paired nonparametric bootstrap interval across seeds 17, 42, and 97.

The current fuzzy control records precision 0.833333, recall 0.625, and F1 0.714286 on this small fixture. CI requires at least 0.8 precision and 0.5 recall. Those numbers verify a controlled detector path only. They are not production prevalence estimates.

Evaluation statistics

The earlier function was named paired bootstrap, but its implementation randomly flipped paired differences. That procedure is a paired randomization test. The rebuild names it correctly and separates it from the bootstrap interval.

Statistical significance on internally generated references remains an internal regression signal. It is not a substitute for an independent benchmark or human review.

Failure semantics

Condition	Behaviour
Any declared stage fails	Failure event is written and the pipeline exits nonzero
Cached output is missing or modified	Cache entry is rejected and the stage executes again
Benchmark missing or empty	Contamination stage exits nonzero
Contamination threshold crossed	Report is written, then pipeline exits nonzero
Candidate source rights are unknown	Source governance and release creation fail
Training use is not permitted	Source is quarantined and the gate fails
Schema or privacy control fails	Record is quarantined with machine-readable reason codes
Source provenance missing	Split refuses to run
Release artifact hash disagrees	Release creation or independent verification fails

Verified development path

CI uses pinned direct development dependencies and Python 3.11. It checks Ruff linting, Ruff formatting, Mypy, Python syntax, Pytest, a 70 percent coverage threshold, a labelled curation calibration gate, a blind evaluation-control gate, and a repeatable release smoke test.

make lint
make typecheck
make test
make coverage
make curation-calibration
make evaluation-fixture

The offline smoke command completes all 12 typed stages, including both governance boundaries, exact and fuzzy deduplication, lexical and fuzzy contamination, dataset profiling, and independent release verification. CI runs it twice and requires 12 verified cache hits plus an unchanged release identity on the second run. Semantic controls are intentionally marked as not run in the smoke release.

What remains before a model result is published

1. Construct an independently authored evaluation set with documented provenance.
2. Collect at least 200 genuine reviews, including 50 overlapping items, under the frozen protocol.
3. Run the versioned different-family judge and publish its agreement and reversed-order consistency.
4. Run multiple training seeds or publish an equivalent uncertainty design.
5. Tag the release only after a fresh-install reproduction and green CI.

---

See the evaluation controls, the source and tests, or return to the plain-language overview.