ISO/IEC 27001:2022 vendor evaluation criteria for enterprise AI implementation — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · ISO/IEC 27001:2022 · layout control_review · implementation executed in CI before publish

The control that resolves this is a Gated Vendor Evaluation Framework, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is ISO/IEC 27001:2022, and the question it actually asks is is this risk on the register with a named owner and a treatment. The specific problem here is RFP and structured evaluation. The shape below holds p95 ≤ 600ms over outbound webhooks with signed payloads, and ships with a standard-library implementation that runs as written.

The control gap

A vendor comparison built as a weighted average lets a supplier average its way past a control it does not have. The scorecard ranks everyone, the committee cannot see which gates were failed, and the decision gets re-litigated after the contract is signed.

What makes RFP and structured evaluation specifically hard

An RFP scored as a weighted average lets a vendor average its way past a control it does not have. The scoring model, not the questions, is usually what makes the evaluation indefensible.

What resolves it is a two-layer scorecard: pass/fail gates, then weights. The failure to design against: the highest-scoring vendor failing security review after award.

What ISO/IEC 27001:2022 specifically requires

ISO 27001 is a management-system standard: the auditor checks that risks are identified, owned, treated and reviewed on a cycle. A perfect technical control that is absent from the risk register is a nonconformity.

The artifact that satisfies it is risk register entries with owners, treatments and review dates. The way teams most often fail it: a technically sound control that no Statement of Applicability references, so it cannot be evidenced.

Risk register with acceptance criteria

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Deployment parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

What the auditor will ask

• Where is this control in the risk register and the Statement of Applicability, who owns it, and when was it last reviewed?
  Register entries are generated from the deployed control set, so every control has an SoA reference, a named owner and a review date that cannot silently lapse.
• Which criteria are pass/fail, and which vendor failed one?
  Gates are evaluated first and independently of score: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, residency. A failure on any of them disqualifies.
• Which criteria are pass/fail, and which vendor failed one?
  The gate layer is pass/fail: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, and data residency. Gates are evaluated before any weighted scoring, so a vendor cannot average past a control it lacks.
• For each passed gate, what artifact proves it and what is its date?
  Each gate names the artifact that satisfies it — the report, the signed DPA, the architecture document — with its date. A claim is not evidence.
• Is the latency target in the signed agreement with a remedy, or on a status page?
  It must be in the signed agreement with a remedy. A status page is not a commitment; the scorecard scores the remedy, not the promise.
• What does exit look like — specifically, what can you export and has it been rehearsed?
  Export of prompts, embeddings, eval sets and fine-tunes, rehearsed during the pilot rather than promised in the contract. Portability that has never been executed is not portability.
• Was the reference deployment in the same vertical and regulatory regime as ours?
  It must be — a reference in the same vertical and the same regulatory regime, worked through a structured question set, not a vendor-selected happy customer.

System architecture

Gated Vendor Evaluation Framework — the request path, with the decision points that must exist before the first model call:

flowchart TB
    V[Vendor responses] --> G{Gates - pass/fail}
    G -->|any gate failed| DQ[Disqualified<br/>with named gates]
    G -->|all gates passed| W[Weighted scoring]
    W --> TCO[3-year fully loaded TCO]
    W --> REF[Reference call<br/>comparable regulated deployment]
    TCO --> R[Ranking]
    REF --> R
    R --> DR[Decision record<br/>score + reason per criterion]
    DQ --> DR
    DR --> SC[Steering committee]

The parts you have to build

1. Gate layer — Pass/fail controls: SOC 2 Type 2 currency, DPA, tenant isolation, residency.
2. Weighted layer — Ranks only vendors that cleared every gate.
3. Evidence requirement — Each gate names the artifact that satisfies it, not a claim.
4. TCO model — Fully loaded three-year: licence, run cost, internal effort.
5. Reference protocol — A structured call with a comparable regulated deployment.
6. Decision record — Ranking plus failed gates plus a written reason per score.

Implementation artifact — Weighted scorecard with disqualifying gates

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind outbound webhooks with signed payloads; the failure mode it guards against is at-least-once delivery handled as exactly-once by the consumer. The output you hand the assessor is the Statement of Applicability extract.

"""Weighted vendor scorecard with disqualifying gates.

Why this shape: a procurement scorecard that is a weighted average lets a vendor
average its way past a control it simply does not have. Real evaluations have
two layers — GATES that are pass/fail, and weights that only ever rank the
vendors that already cleared every gate.

This produces the artifact a steering committee signs: a ranking, the gates each
vendor failed, and a written reason per score.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Criterion:
    key: str
    weight: float          # relative; normalised at scoring time
    gate: bool = False     # True -> pass/fail, cannot be averaged away
    note: str = ""


DEFAULT_CRITERIA = [
    Criterion("soc2_type2_current", 0.0, gate=True,
              note="Type 2 report inside 12 months, not a Type 1 or a roadmap"),
    Criterion("dpa_and_subprocessors", 0.0, gate=True,
              note="DPA signable as-is and a disclosed subprocessor list"),
    Criterion("tenant_isolation_evidence", 0.0, gate=True,
              note="Architectural isolation, not a WHERE clause"),
    Criterion("data_residency", 0.0, gate=True,
              note="Processing stays in the contracted region"),
    Criterion("exit_and_portability", 2.0,
              note="Export of prompts, embeddings, evals; no model lock-in"),
    Criterion("latency_slo_contractual", 3.0,
              note="p95 in the agreement with a remedy, not a status page"),
    Criterion("integration_depth", 3.0,
              note="Works against the real system of record, not a demo schema"),
    Criterion("implementation_capacity", 2.0,
              note="Named engineers, not a bench claim"),
    Criterion("tco_3yr", 3.0, note="Fully loaded: licence, run, internal effort"),
    Criterion("reference_in_our_vertical", 2.0,
              note="A reference call with a comparable regulated deployment"),
]


class Disqualified(Exception):
    pass


@dataclass
class Vendor:
    name: str
    gates: dict = field(default_factory=dict)    # key -> bool
    scores: dict = field(default_factory=dict)   # key -> 0..5
    reasons: dict = field(default_factory=dict)


def evaluate(vendors, criteria=None):
    """-> (ranked, disqualified). Gates are evaluated first, always."""
    criteria = criteria or DEFAULT_CRITERIA
    gate_keys = [c.key for c in criteria if c.gate]
    weighted = [c for c in criteria if not c.gate and c.weight > 0]
    total_w = sum(c.weight for c in weighted) or 1.0

    ranked, disqualified = [], []
    for v in vendors:
        failed = [k for k in gate_keys if not v.gates.get(k, False)]
        if failed:
            disqualified.append({"vendor": v.name, "failed_gates": failed})
            continue
        got = 0.0
        breakdown = []
        for c in weighted:
            s = float(v.scores.get(c.key, 0))
            if not 0 <= s <= 5:
                raise ValueError(f"{v.name}.{c.key}={s} outside 0..5")
            contribution = (c.weight / total_w) * (s / 5.0)
            got += contribution
            breakdown.append({"criterion": c.key, "score": s,
                              "weight": c.weight,
                              "contribution": round(contribution, 4),
                              "reason": v.reasons.get(c.key, ""),
                              "standard": c.note})
        ranked.append({"vendor": v.name, "score": round(got * 100, 1),
                       "breakdown": breakdown})
    ranked.sort(key=lambda r: -r["score"])
    return ranked, disqualified


if __name__ == "__main__":
    passing = {k: True for k in
               ("soc2_type2_current", "dpa_and_subprocessors",
                "tenant_isolation_evidence", "data_residency")}
    a = Vendor("Vendor A", gates=dict(passing),
               scores={"exit_and_portability": 4, "latency_slo_contractual": 5,
                       "integration_depth": 4, "implementation_capacity": 3,
                       "tco_3yr": 3, "reference_in_our_vertical": 5},
               reasons={"latency_slo_contractual": "p95 450ms with 10% credit"})
    b = Vendor("Vendor B", gates={**passing, "tenant_isolation_evidence": False},
               scores={k: 5 for k in
                       ("exit_and_portability", "latency_slo_contractual",
                        "integration_depth", "implementation_capacity",
                        "tco_3yr", "reference_in_our_vertical")})
    ranked, dq = evaluate([a, b])
    for r in ranked:
        print(f"{r['vendor']}: {r['score']}")
    print("disqualified despite perfect scores:", dq)

The ISO/IEC 27001:2022 control, as code

Statement of Applicability with operating evidence. This is the artifact an assessor asks for under ISO/IEC 27001:2022 specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Annex A control to evidence mapping behind a Statement of Applicability.

ISO/IEC 27001:2022 turns on the SoA: every Annex A control is either applicable
with a justification and evidence, or excluded with a justification. The audit
failure is an applicable control whose evidence is a policy document rather
than a record that the control operated.
"""

ANNEX_A_2022 = {
    "A.5.23": "Information security for use of cloud services",
    "A.8.2": "Privileged access rights",
    "A.8.10": "Information deletion",
    "A.8.12": "Data leakage prevention",
    "A.8.15": "Logging",
    "A.8.16": "Monitoring activities",
    "A.8.24": "Use of cryptography",
    "A.8.28": "Secure coding",
}

EVIDENCE_KINDS = {"record", "configuration", "test_result"}
NON_EVIDENCE = {"policy", "intent", "roadmap"}


class StatementOfApplicability:
    def __init__(self, scope: str):
        self.scope = scope
        self.entries: dict[str, dict] = {}

    def apply(self, control: str, justification: str,
              evidence_ref: str, evidence_kind: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if evidence_kind in NON_EVIDENCE:
            raise ValueError(
                f"{control}: {evidence_kind!r} describes intent, not operation")
        if evidence_kind not in EVIDENCE_KINDS:
            raise ValueError(f"{control}: unknown evidence kind {evidence_kind!r}")
        self.entries[control] = {"applicable": True, "justification": justification,
                                 "evidence_ref": evidence_ref,
                                 "evidence_kind": evidence_kind}

    def exclude(self, control: str, justification: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if len(justification.split()) < 5:
            raise ValueError(f"{control}: exclusion needs a real justification")
        self.entries[control] = {"applicable": False, "justification": justification}

    def unaddressed(self) -> list[str]:
        return sorted(set(ANNEX_A_2022) - set(self.entries))

    def certifiable(self) -> dict:
        gaps = self.unaddressed()
        applied = [c for c, e in self.entries.items() if e["applicable"]]
        return {"scope": self.scope, "applied": sorted(applied),
                "excluded": sorted(c for c, e in self.entries.items()
                                   if not e["applicable"]),
                "unaddressed": gaps, "certifiable": not gaps}


if __name__ == "__main__":
    soa = StatementOfApplicability("LLM platform, production tenant")
    soa.apply("A.5.23", "Model inference is a cloud service",
              "cfg://gateway/allowed-providers.yaml", "configuration")
    soa.apply("A.8.2", "Operators hold privileged access to prompts",
              "rec://iam/quarterly-access-review-2026Q3", "record")
    soa.apply("A.8.10", "Retention is enforced on prompt logs",
              "test://retention/expiry-suite-2026-10", "test_result")
    soa.apply("A.8.12", "Egress to model providers can leak data",
              "cfg://dlp/egress-classifier.yaml", "configuration")
    soa.apply("A.8.15", "All inference events are logged",
              "rec://audit/chain-head-2026-10-16", "record")
    soa.apply("A.8.16", "Anomalous usage is monitored",
              "rec://siem/llm-usage-alerts", "record")
    soa.apply("A.8.24", "TLS and at-rest encryption",
              "test://crypto/tls-config-scan", "test_result")
    soa.exclude("A.8.28", "No application code is developed within this scope "
                          "boundary; the platform is configured, not coded")

    report = soa.certifiable()
    assert report["certifiable"], report
    assert len(report["applied"]) == 7 and len(report["excluded"]) == 1, report

    try:
        soa.apply("A.8.15", "we have a logging policy",
                  "doc://policies/logging.pdf", "policy")
        raise AssertionError("accepted a policy document as operating evidence")
    except ValueError:
        pass

    partial = StatementOfApplicability("partial")
    partial.apply("A.8.15", "logging", "rec://x", "record")
    assert not partial.certifiable()["certifiable"]
    assert len(partial.certifiable()["unaddressed"]) == 7

    print("ISO/IEC 27001:2022 SoA evidence mapping: OK")
    print(f"  applied {len(report['applied'])}, excluded {len(report['excluded'])}, "
          f"unaddressed {len(report['unaddressed'])}")

Related architectures

• Part of enterprise ai vendor evaluation.
• The same control appears in PCI DSS 4.0 vendor evaluation criteria for enterprise AI implementation.
• A different failure mode of the same component: GDPR vendor evaluation criteria for enterprise AI implementation.
• Where this interacts with another regime, see HIPAA vendor evaluation criteria for enterprise AI implementation.
• Where this interacts with another regime, see SOX vendor evaluation criteria for enterprise AI implementation.
• If this is the blocker in front of you: enterprise AI infrastructure review.