Evaluation
A model has to earn its release
Most AI tools ask you to trust a demo. At Haryts, every new model version must pass a release gate — safety first, then no regression — before it reaches you. We'll publish measured results once the next model version clears that gate.
How we measure
A release gate, and the two tools behind it
The release gate
A new model version reaches production only after it passes every check, in order: safety first — zero confirmed failures on manipulation and cross-company tests; no regression on protected categories; pre-registered targets, met with a statistical test; then operational checks.
Candidates that failed the gate have been held back; the current version stays in production until a successor passes.
HarytsBench
An internal benchmark over fixed reference corpora. Every model candidate is run against it, and its results feed the release gate.
Internal benchmark; a public benchmark is on the roadmap.
Haryts Eval
An evaluation runner for answer and retrieval quality. In a pilot, it measures quality on your own documents — so you judge Haryts on your knowledge, not on a generic score.
Run on your documents during a pilot.
Why a gate, not a headline number
A model that hasn't earned it doesn't ship
A single average score can hide a regression. So the targets are fixed before testing, every candidate is tested against them, and a model that misses is held back — even if it looks better on average.
- 1Targets are registered before a candidate is tested.
- 2Safety comes first: manipulation attempts and cross-company questions must produce zero confirmed failures.
- 3Protected categories are compared with the model in production: no regression is allowed.
- 4Targets are checked with a statistical test, then operational checks — only then is a release decision made.
FAQ
Evaluation questions
How does Haryts measure quality?
With a release gate that every new model version must pass — safety first, no regression on protected categories, pre-registered targets met with a statistical test, and operational checks — and two internal tools behind it: HarytsBench, a benchmark over fixed reference corpora, and Haryts Eval, which measures answer and retrieval quality.
Why don't you publish a score?
The figures we published before described a model that is no longer in production. We'll publish measured results — with their caveats — once the next model version clears the release gate.
Can we measure Haryts on our own documents?
Yes — that's what a pilot is for. Haryts Eval measures answer and retrieval quality on your documents, against success criteria we agree with you before we start.
Measure Haryts on your own documents
A pilot ends with a written report: validation results, an evaluation baseline, and your knowledge gaps.