AI risk scoring
you can explain
A score that sorts your review queue is genuinely useful. A score nobody can explain is a liability, and in regulated work it is often not permitted. Here is how to build the first kind.
Accuracy is the easy requirement. Justifying the score is the hard one.
Risk scoring is one of the oldest and most valuable applications of machine learning. You have more cases than reviewers, so you rank them and work down the list. Fraud, claims, onboarding, credit, compliance alerts: the shape is the same everywhere.
What has changed is the standard of proof. It is no longer enough for the score to work. You need to explain, for an individual case, why it scored the way it did. Your reviewer needs it to do their job, your auditor needs it to sign off, and increasingly the person being scored has a right to it.
That constraint should shape the design from day one. Retrofitting explainability onto an opaque model is expensive and rarely convincing.
Five principles for scoring that survives scrutiny.
Score to prioritise, not to decide
The single most important design decision, and the one that determines how much regulatory weight the system has to carry.
- The score orders the queue. Its job is to put the cases most likely to need attention in front of a person first. Framed this way, being imperfect is acceptable, because a person still decides.
- The decision stays with a named role. Someone accountable makes the call, with the score as one input among several. This is both good practice and, for high-risk uses under the EU AI Act, an obligation.
- Avoid automatic adverse outcomes. A score that on its own declines, suspends or rejects someone raises the compliance burden sharply and is rarely worth the efficiency gained.
- Record the human override. When a reviewer disagrees with the score, capture that. It is your best evaluation data and your evidence that oversight is real rather than nominal.
Build from signals a person recognises
Explainability is much easier to achieve by choosing interpretable inputs than by adding an explanation layer on top of something opaque.
- Name your signals in domain language. 'Address changed within 30 days of application' is a signal an expert and an auditor both understand. An unnamed embedding dimension is not.
- Show per-case contributions. For each scored case, display which signals pushed it up and by how much. This is what a reviewer actually needs, and what turns a number into a decision aid.
- Prefer simpler models where they suffice. A well-built interpretable model is often close in performance and vastly easier to defend. Reach for complexity only when the gain is measured and material.
- Write down what the score is not. Documenting the limitations, known blind spots and out-of-scope uses protects both your reviewers and you. It is also expected in any serious audit.
Testing for discrimination properly
Not using a protected attribute proves nothing. Proxies are everywhere, and the only way to know is to measure outcomes.
Measure outcomes across groups
Compare score distributions and downstream decisions across the groups that matter for your context. Do this before launch, with real historical data.
Hunt for proxies deliberately
Postcode, name, device, tenure and channel can all stand in for something you never intended to score on. Check each input for what it might be encoding.
Set a threshold for concern in advance
Decide what disparity would trigger investigation before you see the results. Deciding afterwards invites rationalisation.
Document the tests and repeat them
Fairness is not a launch checkbox. Schedule the same tests, keep the results, and treat a change in them as an incident worth explaining.
Catching drift before someone else does
Scoring systems do not crash. They get gradually less useful while continuing to output plausible numbers, which is far harder to notice.
- Watch the score distribution. A shift in the shape of your scores is usually the first visible sign that something upstream changed. Alert on it.
- Track outcomes, not just scores. Compare what the score predicted against what actually happened, on a rolling basis. Without an outcome loop you are monitoring the weather forecast and never looking outside.
- Monitor override rates by reviewer. A rising override rate means your reviewers have stopped trusting the score, and they usually notice degradation before your dashboards do.
- Version everything. Model version, signal definitions, thresholds, and which version scored each case. When someone asks about a decision from eight months ago, this is the only way to answer.
Where the regulation bites
If your scoring touches individuals in the EU, assume obligations apply and check rather than assume they do not.
- Classify the system before you build. Whether your use falls into the EU AI Act's high-risk category changes the documentation, oversight and testing you owe. Finding out late is expensive.
- Human oversight has to be meaningful. A reviewer who rubber-stamps a hundred scores an hour is not oversight. The workload and the interface have to make genuine review possible.
- Keep the technical documentation current. Data sources, design choices, evaluation results, known limitations, monitoring. Written as you go, because reconstructing it later is far harder.
- Be able to explain an individual case. Not the model in general, but why this person on this date received this score. Design for that question, because eventually someone will ask it.
Risk Brain is our version of this.
We build scoring systems designed for explanation and oversight from the start. Our EU AI Act guide covers the compliance side in detail. Tell us what you need to prioritise and we will tell you what a defensible version looks like. Get in touch.