What Forge does, and what it does not claim.
A training data pipeline should make it easy to inspect where examples came from, what crossed each quality gate, and how an evaluation was constructed.
The problem
A typical synthetic data demo ends after generation. It may show plausible examples, but it does not answer whether the same source leaked across train and test, whether evaluation material appeared in training, or whether a reported improvement survives basic uncertainty analysis.
Forge treats those questions as pipeline requirements. It creates training examples, but the repository is organised around traceability, usage rights, privacy decisions, duplicate controls, and failure conditions rather than a single attractive metric.
The workflow
1. Identify the source
Each document and chunk receives a deterministic content-derived identifier before examples are generated.
2. Establish permitted use
Candidate builds require a source policy. Unknown rights or disallowed training use prevent release.
3. Generate and curate
Schema, structured privacy, exact, fuzzy, semantic, rubric, and difficulty controls add inspectable decisions.
4. Check the benchmark boundary
Lexical, fuzzy, and candidate semantic controls compare generated records with independent benchmark text.
5. Split whole sources
Examples from one source document stay together. The algorithm records the achieved whole-source fraction.
6. Profile and release
Configuration, audit decisions, quarantine files, profile slices, hashes, overlap checks, and gates stay with the release.
How to read the evidence
| Artifact | What it supports | What it does not support |
|---|---|---|
| Smoke run | The pipeline can execute without paid APIs | Model quality or production readiness |
| Split manifest | No shared source or chunk identifiers | Semantic independence between related documents |
| Source governance | Declared license and permitted use are present | Independent legal verification of ownership |
| Structured privacy scan | Configured patterns and checksums were applied | Detection of every personal or contextual identifier |
| MinHash and Jaccard | Measured exact and near-duplicate control behaviour | Semantic equivalence between all records |
| Embedding screen | Similarity under one recorded model and threshold | Infallible paraphrase or contamination detection |
| Model judge | A repeatable rubric-based signal | A replacement for human assessment |
| Controlled evaluation fixture | Blind packets, agreement analysis, and bias gates execute | Genuine human preference or model improvement |
| Paired statistics | Uncertainty on one evaluation set | General capability or causal proof |
Why the historical result is not the headline
The earlier experiment generated references through the same broad process used to create training examples, and the generator and judge were not sufficiently independent. A source-safe split improves the design, but it does not turn an internally generated reference set into an external benchmark. The numbers remain useful for regression analysis, not for claiming general improvement.
Who this is for
Forge is useful for engineers exploring source-grounded task generation, evaluation integrity, reproducible ML workflows, and local fine-tuning. It is also a compact reference for discussing leakage, contamination, judge bias, provenance, and uncertainty in an interview.
Continue to the evaluation design, the technical implementation, or inspect the repository and tests.