ISO/IEC 27001:2022 reference architecture for enterprise LLM deployment at scale — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · ISO/IEC 27001:2022 · layout diagram_first · implementation executed in CI before publish

The control that resolves this is a Tenant-Fair Inference Tier, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is ISO/IEC 27001:2022, and the question it actually asks is is this risk on the register with a named owner and a treatment. The shape below holds p95 ≤ 450ms over durable queue (SQS / Kafka) with an outbox, and ships with a standard-library implementation that runs as written.

Reference architecture

Tenant-Fair Inference Tier — the request path, with the decision points that must exist before the first model call:

flowchart TB
    subgraph Tenants
      TA[Tenant A - batch]
      TB[Tenant B - interactive p95 400ms]
    end
    TA --> AD{Admission}
    TB --> AD
    AD -->|bucket empty| S429[429 + Retry-After]
    AD -->|ceiling full,<br/>outside reservation| S429
    AD -->|admit| Q[Deadline queue]
    Q -->|deadline passed| SH[Shed - do not serve late]
    Q --> W[Worker pool]
    W --> P[Model / vLLM]
    P --> OBS[Per-tenant p95 / p99 window]
    OBS -->|breach| ALERT[Page + autoscale signal]

The parts you have to build

1. Per-tenant token bucket — Sustained rate plus burst, so one tenant cannot spend another's budget.
2. Reservation ledger — Guaranteed in-flight slots per tenant; reservations must sum below the ceiling.
3. Global concurrency ceiling — Bounds the shared tier so it degrades predictably instead of collapsing.
4. Percentile window — Streaming p95/p99 per tenant — the number the contract is written on.
5. Queue with deadline — Requests carry a deadline and are shed rather than served late.
6. Shed path — 429 with an accurate Retry-After so clients back off instead of hot-looping.

Enterprise implementation parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The control gap

A shared inference tier holds under average load and breaches its p95 under a neighbour's batch job. The SLA is written on a percentile, the dashboards show a mean, and admission control is a single global rate limit that protects the provider rather than any individual tenant.

What ISO/IEC 27001:2022 specifically requires

ISO 27001 is a management-system standard: the auditor checks that risks are identified, owned, treated and reviewed on a cycle. A perfect technical control that is absent from the risk register is a nonconformity.

The artifact that satisfies it is risk register entries with owners, treatments and review dates. The way teams most often fail it: a technically sound control that no Statement of Applicability references, so it cannot be evidenced.

Implementation artifact — Per-tenant admission control and percentile tracking

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind durable queue (SQS / Kafka) with an outbox; the failure mode it guards against is a consumer crash between side effect and acknowledgement. The output you hand the assessor is the Statement of Applicability extract.

"""Per-tenant admission control for a shared inference tier.

Why this shape: a global rate limit protects the provider and nobody else. The
failure enterprises actually hit is the noisy-neighbour one — tenant A's batch
job saturates the queue and tenant B breaches the p95 latency SLA that is
written into their contract. You cannot fix that downstream of admission.

Two controls, both required:
  * a per-tenant token bucket, so one tenant cannot consume another's budget
  * a global concurrency ceiling with per-tenant reservations, so the shared
    tier degrades predictably instead of collapsing
"""
import heapq
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    rate: float            # sustained requests per second
    burst: float           # instantaneous allowance
    tokens: float = None
    updated: float = field(default_factory=time.monotonic)

    def __post_init__(self):
        if self.rate <= 0 or self.burst <= 0:
            raise ValueError("rate and burst must be positive")
        if self.tokens is None:
            self.tokens = self.burst

    def take(self, n: float = 1.0, now: float = None) -> bool:
        now = time.monotonic() if now is None else now
        elapsed = max(0.0, now - self.updated)
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
        self.updated = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

    def retry_after(self, n: float = 1.0) -> float:
        """Seconds until n tokens exist. Return this in a 429 Retry-After so
        clients back off correctly instead of hot-looping."""
        deficit = max(0.0, n - self.tokens)
        return deficit / self.rate


class Admission:
    """Per-tenant buckets plus a global concurrency ceiling with reservations.

    reserved: tenant -> guaranteed in-flight slots. The sum of reservations
    must stay below `ceiling`, or the guarantee is fiction.
    """

    def __init__(self, ceiling: int, reserved: dict = None):
        self.ceiling = ceiling
        self.reserved = dict(reserved or {})
        if sum(self.reserved.values()) > ceiling:
            raise ValueError(
                f"reservations {sum(self.reserved.values())} exceed ceiling "
                f"{ceiling}: at least one tenant's SLA cannot be honoured")
        self.buckets: dict = {}
        self.inflight: dict = {}

    def configure(self, tenant: str, rate: float, burst: float):
        self.buckets[tenant] = TokenBucket(rate=rate, burst=burst)

    def admit(self, tenant: str, cost: float = 1.0) -> tuple:
        """-> (admitted, reason, retry_after_seconds)"""
        bucket = self.buckets.get(tenant)
        if bucket is None:
            return False, "tenant_not_provisioned", None

        used = sum(self.inflight.values())
        mine = self.inflight.get(tenant, 0)
        my_reservation = self.reserved.get(tenant, 0)

        # A tenant inside its reservation is admitted even when the shared pool
        # is full — that is what makes the reservation worth signing.
        if used >= self.ceiling and mine >= my_reservation:
            return False, "global_ceiling", 0.05

        if not bucket.take(cost):
            return False, "tenant_rate_limit", round(bucket.retry_after(cost), 3)

        self.inflight[tenant] = mine + 1
        return True, "ok", None

    def release(self, tenant: str):
        self.inflight[tenant] = max(0, self.inflight.get(tenant, 0) - 1)


class LatencyWindow:
    """Streaming p95/p99 over a bounded window.

    An average hides exactly the breach you are contractually exposed to: an
    SLA is written on a percentile, so the number you alert on must be one.
    """

    def __init__(self, size: int = 2048):
        self.size = size
        self.samples: list = []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        ordered = sorted(self.samples)
        # Nearest-rank: the smallest sample at or above the q-th position.
        idx = min(len(ordered) - 1, max(0, int(round(q * len(ordered) + 0.5)) - 1))
        return ordered[idx]

    def breach(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.quantile(q) > slo_ms


if __name__ == "__main__":
    adm = Admission(ceiling=10, reserved={"bank-eu": 4})
    adm.configure("bank-eu", rate=5, burst=5)
    adm.configure("retail-us", rate=50, burst=50)
    for _ in range(9):
        adm.admit("retail-us")
    print("noisy neighbour holds", adm.inflight)
    print("reserved tenant still admitted:", adm.admit("bank-eu"))
    print("burst exhausted ->", [adm.admit("bank-eu") for _ in range(5)][-1])

    win = LatencyWindow()
    for ms in [120, 140, 135, 128, 900, 131, 139, 142, 150, 2200]:
        win.observe(ms)
    print(f"p95={win.quantile(0.95):.0f}ms breach(400ms)={win.breach(400)}")

The ISO/IEC 27001:2022 control, as code

Statement of Applicability with operating evidence. This is the artifact an assessor asks for under ISO/IEC 27001:2022 specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Annex A control to evidence mapping behind a Statement of Applicability.

ISO/IEC 27001:2022 turns on the SoA: every Annex A control is either applicable
with a justification and evidence, or excluded with a justification. The audit
failure is an applicable control whose evidence is a policy document rather
than a record that the control operated.
"""

ANNEX_A_2022 = {
    "A.5.23": "Information security for use of cloud services",
    "A.8.2": "Privileged access rights",
    "A.8.10": "Information deletion",
    "A.8.12": "Data leakage prevention",
    "A.8.15": "Logging",
    "A.8.16": "Monitoring activities",
    "A.8.24": "Use of cryptography",
    "A.8.28": "Secure coding",
}

EVIDENCE_KINDS = {"record", "configuration", "test_result"}
NON_EVIDENCE = {"policy", "intent", "roadmap"}


class StatementOfApplicability:
    def __init__(self, scope: str):
        self.scope = scope
        self.entries: dict[str, dict] = {}

    def apply(self, control: str, justification: str,
              evidence_ref: str, evidence_kind: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if evidence_kind in NON_EVIDENCE:
            raise ValueError(
                f"{control}: {evidence_kind!r} describes intent, not operation")
        if evidence_kind not in EVIDENCE_KINDS:
            raise ValueError(f"{control}: unknown evidence kind {evidence_kind!r}")
        self.entries[control] = {"applicable": True, "justification": justification,
                                 "evidence_ref": evidence_ref,
                                 "evidence_kind": evidence_kind}

    def exclude(self, control: str, justification: str) -> None:
        if control not in ANNEX_A_2022:
            raise KeyError(f"{control} is not an Annex A:2022 control")
        if len(justification.split()) < 5:
            raise ValueError(f"{control}: exclusion needs a real justification")
        self.entries[control] = {"applicable": False, "justification": justification}

    def unaddressed(self) -> list[str]:
        return sorted(set(ANNEX_A_2022) - set(self.entries))

    def certifiable(self) -> dict:
        gaps = self.unaddressed()
        applied = [c for c, e in self.entries.items() if e["applicable"]]
        return {"scope": self.scope, "applied": sorted(applied),
                "excluded": sorted(c for c, e in self.entries.items()
                                   if not e["applicable"]),
                "unaddressed": gaps, "certifiable": not gaps}


if __name__ == "__main__":
    soa = StatementOfApplicability("LLM platform, production tenant")
    soa.apply("A.5.23", "Model inference is a cloud service",
              "cfg://gateway/allowed-providers.yaml", "configuration")
    soa.apply("A.8.2", "Operators hold privileged access to prompts",
              "rec://iam/quarterly-access-review-2026Q3", "record")
    soa.apply("A.8.10", "Retention is enforced on prompt logs",
              "test://retention/expiry-suite-2026-10", "test_result")
    soa.apply("A.8.12", "Egress to model providers can leak data",
              "cfg://dlp/egress-classifier.yaml", "configuration")
    soa.apply("A.8.15", "All inference events are logged",
              "rec://audit/chain-head-2026-10-16", "record")
    soa.apply("A.8.16", "Anomalous usage is monitored",
              "rec://siem/llm-usage-alerts", "record")
    soa.apply("A.8.24", "TLS and at-rest encryption",
              "test://crypto/tls-config-scan", "test_result")
    soa.exclude("A.8.28", "No application code is developed within this scope "
                          "boundary; the platform is configured, not coded")

    report = soa.certifiable()
    assert report["certifiable"], report
    assert len(report["applied"]) == 7 and len(report["excluded"]) == 1, report

    try:
        soa.apply("A.8.15", "we have a logging policy",
                  "doc://policies/logging.pdf", "policy")
        raise AssertionError("accepted a policy document as operating evidence")
    except ValueError:
        pass

    partial = StatementOfApplicability("partial")
    partial.apply("A.8.15", "logging", "rec://x", "record")
    assert not partial.certifiable()["certifiable"]
    assert len(partial.certifiable()["unaddressed"]) == 7

    print("ISO/IEC 27001:2022 SoA evidence mapping: OK")
    print(f"  applied {len(report['applied'])}, excluded {len(report['excluded'])}, "
          f"unaddressed {len(report['unaddressed'])}")

Risk mitigation matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Questions to answer before approval

• Where is this control in the risk register and the Statement of Applicability, who owns it, and when was it last reviewed?
  Register entries are generated from the deployed control set, so every control has an SoA reference, a named owner and a review date that cannot silently lapse.
• Which tenants have contractual percentile targets, and what reserves capacity for them?
  Any tenant with a contractual percentile gets an entry in the reservation ledger — guaranteed in-flight slots inside the global ceiling. Reservations are validated at configuration time and must sum below the ceiling.
• What happens to tenant B's p95 when tenant A sends 10x its normal volume — measured, not assumed?
  Tenant B stays inside its SLO, because tenant A is bounded by its own token bucket and cannot consume B's reserved slots. This is testable: saturate A and watch B's p95 window.
• Is the alert on a percentile or a mean?
  A percentile. The SLA is written on p95, so the alert is on p95. The mean is dashboard-only, because it hides exactly the breach you are contractually exposed to.
• What is shed under saturation, and how does a client learn when to retry?
  Requests past their deadline are shed rather than served late, and the shed path returns 429 with an accurate Retry-After computed from the bucket's refill rate so clients back off instead of hot-looping.
• Can a cache entry created for one tenant ever be served to another?
  No. Every cache key is namespaced by tenant id; keying on prompt text alone is the mechanism by which one tenant's completion is served to another.

Adjacent problems

• Part of enterprise llm architecture at scale.
• HIPAA reference architecture for enterprise LLM deployment at scale applies the same mechanism under a different obligation.
• The same control appears in enterprise LLM deployment at scale in fintech.
• A different failure mode of the same component: fedRAMP Moderate reference architecture for enterprise LLM deployment.
• The same control appears in SOX reference architecture for enterprise LLM deployment at scale.
• If this is the blocker in front of you: enterprise AI infrastructure review.