Model Review Policy
This page defines how the AI and machine learning components on the platform are reviewed: who conducts it, on what cadence, by what method, and what happens when a finding is raised.
The Responsible AI Policy establishes the review obligation; this page is its procedure; the Model Review Log is the record that it was carried out.
| Version | 1.0 |
| Effective date | 20 September 2026 |
| Scope | All model and decision components running in production |
| Policy owner | Lunixi engineering and compliance — [email protected] |
| Review | Annually |
Purpose
Continuous monitoring tells you whether a model is behaving as expected. A review asks a different question: is the model still doing the right job?
These are not the same question. A model inside its thresholds may be serving a purpose that no longer holds; an experiment whose measurement looks clean may have been measured on a biased sample. Automated monitoring cannot see these, because it does not know what to look for.
Roles and independence
| Role | Responsibility |
|---|---|
| Reviewer | Conducts the review, classifies findings, writes the record |
| Component owner | Supplies evidence, responds to findings, carries out remediation |
| Approver | Closes the record; decides on intervention for critical findings |
The reviewer cannot be the author of the change under review. Where team size makes that separation impossible for a given review, it is written into the record as a limitation rather than hidden. We do not present a review without independence as if it had it.
Reviews are currently conducted in house. There is no independent third-party audit yet; when one begins it will be defined on this page as a separate type.
Cadence
| Type | Cadence | When |
|---|---|---|
| Model review | Quarterly | On the calendar date |
| Policy review | Annually | On the anniversary of the effective date |
| Event-driven review | On trigger | On the triggers below, within 10 business days |
Event-driven triggers
- A model output's decision authority widens
- A new model or AI surface goes live
- A threshold alert results in a rollback
- The feature schema is raised and model state is reset
- A customer or regulator raises a formal objection to a decision's rationale
Method — step by step
1. Preparation. Scope is fixed: which components, which period, which versions. Open items from the previous record are carried onto the list.
2. Inventory. The models actually running in production, their versions, feature schema versions and decision authority are enumerated. We look at the running configuration, not at declarations — the inventory is derived from the system, not from a document.
3. Measurement review. The shadow and gradual-rollout reports for the period are read. What is checked:
- Whether the sample threshold was passed, and whether no verdict was issued where it was not
- Whether confidence intervals were reported rather than collapsed to a point estimate
- Whether coverage is known, and where unknown, that it was not counted as zero
- Whether comparison was made at the unit the decision is taken in
4. Explainability sampling. Decisions are drawn at random from the period and their rationales examined. What is sought: whether the rationale carries the decision, whether component contributions sum to the gap, and whether indeterminate conditions appear in the rationale.
5. Incident review. Threshold alerts, rollbacks and the responses to them are reviewed. An alert that was raised and left unanswered is itself a finding.
6. Open items. Gaps carried from the previous review are either closed or carried forward with reasons. An item carried forward twice is reclassified one severity level higher.
7. Record. Findings are classified, intervention decisions are taken, and a dated record is written and approved.
Finding classes and response times
| Class | Definition | Response |
|---|---|---|
| Critical | Evidence of a wrong decision on the money or compliance path; a decision that cannot be explained | Immediate — does not wait for the record |
| High | Measurement invalid, coverage unknown, or a gate not actually working | 10 business days |
| Medium | Weak rationale, inventory not matching documentation, an alert left unanswered | By the next review |
| Low | Documentation, naming and traceability improvements | No deadline |
Escalation — on a critical finding
On a critical finding the review does not stop, but intervention does not wait. Our levers, from least to most intrusive:
- Set the gradual-rollout percentage to zero — the candidate model stops affecting traffic; learning continues.
- Roll back to the previous model version — because snapshots are stored with their schema and model version, a rollback preserves learned state rather than starting from zero.
- Republish the previous flow version — on the fraud side a rollback is a new publication, not an edit; history is never rewritten.
- Disable the component — the decision path falls back to defined behaviour without it.
The intervention decision belongs to a human. The threshold monitor performs no automatic rollback. This is deliberate: automatic rollback reacts to measurement noise and destabilises a settled configuration.
Where a critical finding has affected customers, the notification obligation follows the incident notification timeline on the Trust page.
Evidence and records
Each review produces a dated record containing: scope, inventory, the measurement reports examined, the decisions sampled, the findings and their classes, intervention decisions, open items, the reviewer and the approver.
Records are retained and earlier versions are not destroyed — continuity can only be demonstrated through version history.
The summary is public on the Model Review Log. The detailed record is shared under NDA as part of a vendor assessment.
The limit of this policy
Review does not replace monitoring, and monitoring does not replace review. Continuous monitoring is automatic and threshold-based — the shadow experiment, gradual-rollout measurement and a threshold monitor running every five minutes. The review in this policy is periodic and rests on human judgement. We do not present either as evidence of the other.
Related pages
- Responsible AI Policy — the source of the obligation, principles, monitoring
- Model Review Log — the record of reviews carried out
- EU AI Act — risk classification
- Decision Governance — the publish and rollback chain
Questions
For questions on the procedure, a request for a review record, or a vendor assessment form: [email protected]