EU AI Act vendor evaluation criteria for enterprise AI implementation — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · EU AI Act · layout spec_first · implementation executed in CI before publish

Deployment parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The control that resolves this is a Gated Vendor Evaluation Framework, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is EU AI Act, and the question it actually asks is what risk class is this system, and where is the technical documentation. The specific problem here is RFP and structured evaluation. The shape below holds p95 ≤ 600ms over gRPC with mTLS, and ships with a standard-library implementation that runs as written.

The control gap

A vendor comparison built as a weighted average lets a supplier average its way past a control it does not have. The scorecard ranks everyone, the committee cannot see which gates were failed, and the decision gets re-litigated after the contract is signed.

What makes RFP and structured evaluation specifically hard

An RFP scored as a weighted average lets a vendor average its way past a control it does not have. The scoring model, not the questions, is usually what makes the evaluation indefensible.

What resolves it is a two-layer scorecard: pass/fail gates, then weights. The failure to design against: the highest-scoring vendor failing security review after award.

What EU AI Act specifically requires

The EU AI Act regulates the SYSTEM, not the data. The first question is its risk classification, and a high-risk classification pulls in Article 11 technical documentation, Article 12 record-keeping and Article 14 human oversight — obligations that have to be designed in, not attached later.

The artifact that satisfies it is a model card plus Article 12 automatic logs. The way teams most often fail it: a system classified as limited-risk that is actually high-risk because it informs a decision about a person.

Working implementation — Weighted scorecard with disqualifying gates

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind gRPC with mTLS; the failure mode it guards against is a half-open HTTP/2 stream silently dropping the deadline. The output you hand the assessor is the Annex III classification record and model card.

"""Weighted vendor scorecard with disqualifying gates.

Why this shape: a procurement scorecard that is a weighted average lets a vendor
average its way past a control it simply does not have. Real evaluations have
two layers — GATES that are pass/fail, and weights that only ever rank the
vendors that already cleared every gate.

This produces the artifact a steering committee signs: a ranking, the gates each
vendor failed, and a written reason per score.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Criterion:
    key: str
    weight: float          # relative; normalised at scoring time
    gate: bool = False     # True -> pass/fail, cannot be averaged away
    note: str = ""


DEFAULT_CRITERIA = [
    Criterion("soc2_type2_current", 0.0, gate=True,
              note="Type 2 report inside 12 months, not a Type 1 or a roadmap"),
    Criterion("dpa_and_subprocessors", 0.0, gate=True,
              note="DPA signable as-is and a disclosed subprocessor list"),
    Criterion("tenant_isolation_evidence", 0.0, gate=True,
              note="Architectural isolation, not a WHERE clause"),
    Criterion("data_residency", 0.0, gate=True,
              note="Processing stays in the contracted region"),
    Criterion("exit_and_portability", 2.0,
              note="Export of prompts, embeddings, evals; no model lock-in"),
    Criterion("latency_slo_contractual", 3.0,
              note="p95 in the agreement with a remedy, not a status page"),
    Criterion("integration_depth", 3.0,
              note="Works against the real system of record, not a demo schema"),
    Criterion("implementation_capacity", 2.0,
              note="Named engineers, not a bench claim"),
    Criterion("tco_3yr", 3.0, note="Fully loaded: licence, run, internal effort"),
    Criterion("reference_in_our_vertical", 2.0,
              note="A reference call with a comparable regulated deployment"),
]


class Disqualified(Exception):
    pass


@dataclass
class Vendor:
    name: str
    gates: dict = field(default_factory=dict)    # key -> bool
    scores: dict = field(default_factory=dict)   # key -> 0..5
    reasons: dict = field(default_factory=dict)


def evaluate(vendors, criteria=None):
    """-> (ranked, disqualified). Gates are evaluated first, always."""
    criteria = criteria or DEFAULT_CRITERIA
    gate_keys = [c.key for c in criteria if c.gate]
    weighted = [c for c in criteria if not c.gate and c.weight > 0]
    total_w = sum(c.weight for c in weighted) or 1.0

    ranked, disqualified = [], []
    for v in vendors:
        failed = [k for k in gate_keys if not v.gates.get(k, False)]
        if failed:
            disqualified.append({"vendor": v.name, "failed_gates": failed})
            continue
        got = 0.0
        breakdown = []
        for c in weighted:
            s = float(v.scores.get(c.key, 0))
            if not 0 <= s <= 5:
                raise ValueError(f"{v.name}.{c.key}={s} outside 0..5")
            contribution = (c.weight / total_w) * (s / 5.0)
            got += contribution
            breakdown.append({"criterion": c.key, "score": s,
                              "weight": c.weight,
                              "contribution": round(contribution, 4),
                              "reason": v.reasons.get(c.key, ""),
                              "standard": c.note})
        ranked.append({"vendor": v.name, "score": round(got * 100, 1),
                       "breakdown": breakdown})
    ranked.sort(key=lambda r: -r["score"])
    return ranked, disqualified


if __name__ == "__main__":
    passing = {k: True for k in
               ("soc2_type2_current", "dpa_and_subprocessors",
                "tenant_isolation_evidence", "data_residency")}
    a = Vendor("Vendor A", gates=dict(passing),
               scores={"exit_and_portability": 4, "latency_slo_contractual": 5,
                       "integration_depth": 4, "implementation_capacity": 3,
                       "tco_3yr": 3, "reference_in_our_vertical": 5},
               reasons={"latency_slo_contractual": "p95 450ms with 10% credit"})
    b = Vendor("Vendor B", gates={**passing, "tenant_isolation_evidence": False},
               scores={k: 5 for k in
                       ("exit_and_portability", "latency_slo_contractual",
                        "integration_depth", "implementation_capacity",
                        "tco_3yr", "reference_in_our_vertical")})
    ranked, dq = evaluate([a, b])
    for r in ranked:
        print(f"{r['vendor']}: {r['score']}")
    print("disqualified despite perfect scores:", dq)

The EU AI Act control, as code

Annex IV documentation completeness and Article 12 logging. This is the artifact an assessor asks for under EU AI Act specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Annex IV technical documentation completeness + Article 12 logging.

A high-risk system under the EU AI Act needs technical documentation covering
specified elements, and automatic recording of events over the system lifetime.
The failure is rarely refusal to document — it is documentation that is stale,
partial, or cannot be tied to the exact model version that produced an output.

This checks the documentation set against the Annex IV elements and refuses to
mark a release conformant while any element is missing or stale.
"""
from datetime import date, timedelta

ANNEX_IV_ELEMENTS = {
    "general_description": "intended purpose, provider, version",
    "design_specification": "architecture, algorithmic logic, design choices",
    "training_data": "provenance, scope, characteristics, labelling",
    "human_oversight": "measures enabling natural-person oversight",
    "accuracy_metrics": "accuracy, robustness and cybersecurity metrics",
    "risk_management": "risk management system per Article 9",
    "change_log": "changes made through the lifecycle",
    "post_market_plan": "post-market monitoring plan",
}

STALENESS_DAYS = 180


class TechnicalFile:
    def __init__(self, system: str, model_version: str, today: date):
        self.system, self.model_version, self.today = system, model_version, today
        self.elements: dict[str, date] = {}

    def attach(self, element: str, updated: date) -> None:
        if element not in ANNEX_IV_ELEMENTS:
            raise KeyError(f"{element} is not an Annex IV element")
        self.elements[element] = updated

    def missing(self) -> list[str]:
        return sorted(set(ANNEX_IV_ELEMENTS) - set(self.elements))

    def stale(self) -> list[str]:
        cutoff = self.today - timedelta(days=STALENESS_DAYS)
        return sorted(k for k, v in self.elements.items() if v < cutoff)

    def conformant(self) -> dict:
        missing, stale = self.missing(), self.stale()
        return {"system": self.system, "model_version": self.model_version,
                "missing": missing, "stale": stale,
                "conformant": not missing and not stale}


class EventLog:
    """Article 12: records must tie an output to the version that produced it."""

    def __init__(self):
        self.events: list[dict] = []

    def record(self, model_version: str, event: str, ref: str) -> None:
        self.events.append({"model_version": model_version,
                            "event": event, "ref": ref})

    def trace(self, ref: str) -> dict | None:
        for e in self.events:
            if e["ref"] == ref:
                return e
        return None

    def untraceable(self, known_versions: set[str]) -> list[str]:
        return [e["ref"] for e in self.events
                if e["model_version"] not in known_versions]


if __name__ == "__main__":
    today = date(2026, 10, 16)
    tf = TechnicalFile("claims-triage", "v4.2.0", today)
    for element in ANNEX_IV_ELEMENTS:
        tf.attach(element, today - timedelta(days=30))
    assert tf.conformant()["conformant"], tf.conformant()

    partial = TechnicalFile("claims-triage", "v4.3.0", today)
    for element in list(ANNEX_IV_ELEMENTS)[:5]:
        partial.attach(element, today - timedelta(days=10))
    partial.attach("change_log", today - timedelta(days=400))
    report = partial.conformant()
    assert not report["conformant"], report
    assert "post_market_plan" in report["missing"], report
    assert "change_log" in report["stale"], report

    log = EventLog()
    log.record("v4.2.0", "inference", "out-9001")
    log.record("v4.1.7", "inference", "out-8800")
    assert log.trace("out-9001")["model_version"] == "v4.2.0"
    assert log.untraceable({"v4.2.0"}) == ["out-8800"]

    print("EU AI Act Annex IV + Article 12: OK")
    print(f"  complete file conformant={tf.conformant()['conformant']}, "
          f"partial missing {len(report['missing'])}, stale {len(report['stale'])}")

The blueprint

Gated Vendor Evaluation Framework — the request path, with the decision points that must exist before the first model call:

flowchart TB
    V[Vendor responses] --> G{Gates - pass/fail}
    G -->|any gate failed| DQ[Disqualified<br/>with named gates]
    G -->|all gates passed| W[Weighted scoring]
    W --> TCO[3-year fully loaded TCO]
    W --> REF[Reference call<br/>comparable regulated deployment]
    TCO --> R[Ranking]
    REF --> R
    R --> DR[Decision record<br/>score + reason per criterion]
    DQ --> DR
    DR --> SC[Steering committee]

Implementation components

1. Gate layer — Pass/fail controls: SOC 2 Type 2 currency, DPA, tenant isolation, residency.
2. Weighted layer — Ranks only vendors that cleared every gate.
3. Evidence requirement — Each gate names the artifact that satisfies it, not a claim.
4. TCO model — Fully loaded three-year: licence, run cost, internal effort.
5. Reference protocol — A structured call with a comparable regulated deployment.
6. Decision record — Ranking plus failed gates plus a written reason per score.

Risk and control matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Control questions before sign-off

• What is this system's risk classification under Annex III, who decided, and what happens to that classification when the use case changes?
  The classification is recorded against its Annex III category with a named owner, and a material change of use triggers re-classification before the change ships.
• Which criteria are pass/fail, and which vendor failed one?
  Gates are evaluated first and independently of score: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, residency. A failure on any of them disqualifies.
• Which criteria are pass/fail, and which vendor failed one?
  The gate layer is pass/fail: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, and data residency. Gates are evaluated before any weighted scoring, so a vendor cannot average past a control it lacks.
• For each passed gate, what artifact proves it and what is its date?
  Each gate names the artifact that satisfies it — the report, the signed DPA, the architecture document — with its date. A claim is not evidence.
• Is the latency target in the signed agreement with a remedy, or on a status page?
  It must be in the signed agreement with a remedy. A status page is not a commitment; the scorecard scores the remedy, not the promise.
• What does exit look like — specifically, what can you export and has it been rehearsed?
  Export of prompts, embeddings, eval sets and fine-tunes, rehearsed during the pilot rather than promised in the contract. Portability that has never been executed is not portability.
• Was the reference deployment in the same vertical and regulatory regime as ours?
  It must be — a reference in the same vertical and the same regulatory regime, worked through a structured question set, not a vendor-selected happy customer.

See also

• Part of enterprise ai vendor evaluation.
• SOC 2 Type II vendor evaluation criteria for enterprise AI implementation applies the same mechanism under a different obligation.
• A different failure mode of the same component: ISO/IEC 27001:2022 vendor evaluation criteria for enterprise AI.
• For the related constraint, see SOX vendor evaluation criteria for enterprise AI implementation.
• The same control appears in GDPR vendor evaluation criteria for enterprise AI implementation.
• If this is the blocker in front of you: enterprise AI infrastructure review.