AI-Assisted Evaluation Platform
An enterprise evaluation platform where human assessors and an AI pipeline independently assess the same authorised evidence — with calibration analytics on the disagreement, and material decisions owned by people.
The human evaluates first. AI evaluates the same evidence independently. Where they diverge becomes the signal.
Sequence is the core design decision: the human evaluates first, then AI evaluates the same authorised evidence independently. Neither sees the other's result before committing.
Divergence between the two becomes the useful signal — surfacing where the rubric is ambiguous, where an assessor is drifting, or where the AI is unreliable for a given case type.
Calibration analytics measure score spread, inter-rater agreement, AI variance and holdout performance, so consistency is monitored rather than assumed.
The rubric lives as versioned data, not code — no weight or threshold is embedded in the implementation, so evaluation criteria can change under governance without a release.
Structured evaluation at scale suffers from drift: different assessors apply the same rubric differently, and nobody notices until decisions are challenged.
Adding AI naively makes it worse — if assessors see the AI score first, they anchor to it, and you lose the independent human judgement you were trying to support.
What was needed was a way to measure consistency itself, not simply produce more scores faster.
An enterprise evaluation platform where human assessors and an AI pipeline independently assess the same authorised evidence — with calibration analytics on the disagreement, and material decisions owned by people.
Relevant wherever consistent human judgement matters — quality assessment, underwriting, grading, compliance review — and you want AI to improve consistency without quietly replacing the judgement.
Delivery slices
7 complete, foundation to dashboard
Calibration signals
Spread, inter-rater, AI variance, holdout
Rubric
Versioned YAML — no thresholds in code
Architecture
Pure domain layer, zero I/O, fully unit tested
The constraint that shaped it
AI must not anchor the human
If the assessor sees an AI score first, their independent judgement is gone — and with it the ability to measure anything. Enforcing human-first sequencing costs throughput and is the single most important property of the design. It is what makes the disagreement data meaningful.
Rubric as versioned data
No weight or threshold appears in code. Evaluation criteria are governed data with version history — so a rubric change is a reviewable decision rather than a code deploy.
A pure domain layer
Business rules live in a package with no I/O and no database access, fully unit tested. Evaluation logic is the part that must be provably correct, so it was isolated from infrastructure.
AI advisory, humans accountable
The platform never converts an AI score into a decision. Material outcomes carry a named human approver — which is what makes the system usable where outcomes affect people.
- Designed the human-first evaluation model and the calibration approach
- Designed the evaluator workspace and approver experience
- Shaped the architecture separating domain rules from infrastructure
- Defined the AI boundary — advisory, versioned, independently measured
How We Solved the Problem
Evidence ingestion
Authorised evidence is ingested and normalised so human and AI evaluation operate on exactly the same material.
Human evaluation workspace
Assessors evaluate against the rubric in a purpose-built workspace, committing their judgement before any AI output exists for that case.
Independent AI evaluation
A two-stage pipeline with versioned prompts and provider adapters evaluates the same evidence separately, so its result is genuinely independent rather than a confirmation.
Calibration analytics
Spread, inter-rater agreement, AI variance and holdout comparison quantify how consistent evaluation actually is across people and over time.
Decision workflow
Material decisions route to an approver with both evaluations and their divergence visible — AI remains advisory throughout.
Product Designs
These are real product screens from a client engagement. Enter the access code to view them.
NDA Protected
Concept designs available
Evaluator workspace
The assessor evaluates against the rubric before any AI output exists for that case.
NDA Protected
Concept designs available
Independent AI evaluation
AI assesses the same authorised evidence separately — advisory, never deciding.
NDA Protected
Concept designs available
Calibration report
Score spread, inter-rater agreement, AI variance and holdout performance.
NDA Protected
Concept designs available
Decision queue
Material decisions route to a named approver with both evaluations visible.
NDA Protected
Concept designs available
Evaluation pipeline
Cases moving through intake, preparation and assessment.
NDA Protected
Concept designs available
Velocity dashboard
Funnel, bottlenecks and throughput across the evaluation programme.