Reference architecture for enterprise LLM deployment at scale in fintech — Reference Architecture

Reference architecture · published 2026-10-16 · financial services · SOC 2 Type II · layout walkthrough · implementation executed in CI before publish

The control that resolves this is a Tenant-Fair Inference Tier, evaluated before the request reaches a model provider rather than audited afterwards. In financial services the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The shape below holds p95 ≤ 300ms over gRPC with mTLS, and ships with a standard-library implementation that runs as written.

The control gap

A shared inference tier holds under average load and breaches its p95 under a neighbour's batch job. The SLA is written on a percentile, the dashboards show a mean, and admission control is a single global rate limit that protects the provider rather than any individual tenant.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

Implementation components

1. Per-tenant token bucket — Sustained rate plus burst, so one tenant cannot spend another's budget.
2. Reservation ledger — Guaranteed in-flight slots per tenant; reservations must sum below the ceiling.
3. Global concurrency ceiling — Bounds the shared tier so it degrades predictably instead of collapsing.
4. Percentile window — Streaming p95/p99 per tenant — the number the contract is written on.
5. Queue with deadline — Requests carry a deadline and are shed rather than served late.
6. Shed path — 429 with an accurate Retry-After so clients back off instead of hot-looping.

Reference implementation — Per-tenant admission control and percentile tracking

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind gRPC with mTLS; the failure mode it guards against is a half-open HTTP/2 stream silently dropping the deadline. The output you hand the assessor is the observation-window evidence export.

"""Per-tenant admission control for a shared inference tier.

Why this shape: a global rate limit protects the provider and nobody else. The
failure enterprises actually hit is the noisy-neighbour one — tenant A's batch
job saturates the queue and tenant B breaches the p95 latency SLA that is
written into their contract. You cannot fix that downstream of admission.

Two controls, both required:
  * a per-tenant token bucket, so one tenant cannot consume another's budget
  * a global concurrency ceiling with per-tenant reservations, so the shared
    tier degrades predictably instead of collapsing
"""
import heapq
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    rate: float            # sustained requests per second
    burst: float           # instantaneous allowance
    tokens: float = None
    updated: float = field(default_factory=time.monotonic)

    def __post_init__(self):
        if self.rate <= 0 or self.burst <= 0:
            raise ValueError("rate and burst must be positive")
        if self.tokens is None:
            self.tokens = self.burst

    def take(self, n: float = 1.0, now: float = None) -> bool:
        now = time.monotonic() if now is None else now
        elapsed = max(0.0, now - self.updated)
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
        self.updated = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

    def retry_after(self, n: float = 1.0) -> float:
        """Seconds until n tokens exist. Return this in a 429 Retry-After so
        clients back off correctly instead of hot-looping."""
        deficit = max(0.0, n - self.tokens)
        return deficit / self.rate


class Admission:
    """Per-tenant buckets plus a global concurrency ceiling with reservations.

    reserved: tenant -> guaranteed in-flight slots. The sum of reservations
    must stay below `ceiling`, or the guarantee is fiction.
    """

    def __init__(self, ceiling: int, reserved: dict = None):
        self.ceiling = ceiling
        self.reserved = dict(reserved or {})
        if sum(self.reserved.values()) > ceiling:
            raise ValueError(
                f"reservations {sum(self.reserved.values())} exceed ceiling "
                f"{ceiling}: at least one tenant's SLA cannot be honoured")
        self.buckets: dict = {}
        self.inflight: dict = {}

    def configure(self, tenant: str, rate: float, burst: float):
        self.buckets[tenant] = TokenBucket(rate=rate, burst=burst)

    def admit(self, tenant: str, cost: float = 1.0) -> tuple:
        """-> (admitted, reason, retry_after_seconds)"""
        bucket = self.buckets.get(tenant)
        if bucket is None:
            return False, "tenant_not_provisioned", None

        used = sum(self.inflight.values())
        mine = self.inflight.get(tenant, 0)
        my_reservation = self.reserved.get(tenant, 0)

        # A tenant inside its reservation is admitted even when the shared pool
        # is full — that is what makes the reservation worth signing.
        if used >= self.ceiling and mine >= my_reservation:
            return False, "global_ceiling", 0.05

        if not bucket.take(cost):
            return False, "tenant_rate_limit", round(bucket.retry_after(cost), 3)

        self.inflight[tenant] = mine + 1
        return True, "ok", None

    def release(self, tenant: str):
        self.inflight[tenant] = max(0, self.inflight.get(tenant, 0) - 1)


class LatencyWindow:
    """Streaming p95/p99 over a bounded window.

    An average hides exactly the breach you are contractually exposed to: an
    SLA is written on a percentile, so the number you alert on must be one.
    """

    def __init__(self, size: int = 2048):
        self.size = size
        self.samples: list = []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        ordered = sorted(self.samples)
        # Nearest-rank: the smallest sample at or above the q-th position.
        idx = min(len(ordered) - 1, max(0, int(round(q * len(ordered) + 0.5)) - 1))
        return ordered[idx]

    def breach(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.quantile(q) > slo_ms


if __name__ == "__main__":
    adm = Admission(ceiling=10, reserved={"bank-eu": 4})
    adm.configure("bank-eu", rate=5, burst=5)
    adm.configure("retail-us", rate=50, burst=50)
    for _ in range(9):
        adm.admit("retail-us")
    print("noisy neighbour holds", adm.inflight)
    print("reserved tenant still admitted:", adm.admit("bank-eu"))
    print("burst exhausted ->", [adm.admit("bank-eu") for _ in range(5)][-1])

    win = LatencyWindow()
    for ms in [120, 140, 135, 128, 900, 131, 139, 142, 150, 2200]:
        win.observe(ms)
    print(f"p95={win.quantile(0.95):.0f}ms breach(400ms)={win.breach(400)}")

The SOC 2 Type II control, as code

Type II evidence sampling over the observation window. This is the artifact an assessor asks for under SOC 2 Type II specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""SOC 2 Type II evidence sampling over an observation window.

Type II is not "the control exists" — it is "the control operated throughout
the period". An auditor samples days from the window and asks for proof the
control ran on each sampled day. A gap of one day with no evidence is an
exception, so the population must be built from operational records rather
than asserted.
"""
import hashlib
from datetime import date, timedelta


class ObservationWindow:
    """The population an auditor samples from, plus deterministic selection."""

    def __init__(self, start: date, end: date, control_id: str):
        if end < start:
            raise ValueError("window end precedes start")
        self.start, self.end, self.control_id = start, end, control_id
        self._evidence: dict[date, list[str]] = {}

    def record(self, day: date, artifact_ref: str) -> None:
        if not (self.start <= day <= self.end):
            raise ValueError(f"{day} outside observation window")
        self._evidence.setdefault(day, []).append(artifact_ref)

    def population(self) -> list[date]:
        n = (self.end - self.start).days + 1
        return [self.start + timedelta(days=i) for i in range(n)]

    def gaps(self) -> list[date]:
        """Days with no evidence. Each one is a Type II exception."""
        return [d for d in self.population() if not self._evidence.get(d)]

    def sample(self, size: int) -> list[date]:
        """Deterministic, reproducible selection — an auditor must be able to
        re-derive the same sample from the same seed rather than trust ours."""
        pop = self.population()
        size = min(size, len(pop))
        seed = f"{self.control_id}:{self.start}:{self.end}"
        scored = sorted(
            pop, key=lambda d: hashlib.sha256(f"{seed}:{d}".encode()).hexdigest())
        return sorted(scored[:size])

    def evidence_for(self, day: date) -> list[str]:
        return list(self._evidence.get(day, []))

    def attestation(self, size: int = 25) -> dict:
        sampled = self.sample(size)
        missing = [d for d in sampled if not self._evidence.get(d)]
        return {
            "control_id": self.control_id,
            "window": f"{self.start}..{self.end}",
            "population_days": len(self.population()),
            "sampled_days": len(sampled),
            "exceptions": [str(d) for d in missing],
            "opinion": "unqualified" if not missing else "qualified",
        }


if __name__ == "__main__":
    start, end = date(2026, 1, 1), date(2026, 3, 31)
    w = ObservationWindow(start, end, "CC6.1-tenant-isolation")
    for d in w.population():
        w.record(d, f"s3://evidence/{d}/isolation-check.json")
    clean = w.attestation()
    assert clean["opinion"] == "unqualified", clean
    assert clean["population_days"] == 90, clean

    # One missing day is one exception — and it must survive sampling to matter.
    w2 = ObservationWindow(start, end, "CC6.1-tenant-isolation")
    sampled = set(w2.sample(25))
    for d in w2.population():
        if d != sorted(sampled)[0]:
            w2.record(d, "ok")
    assert w2.attestation()["opinion"] == "qualified", w2.attestation()
    assert len(w2.gaps()) == 1, w2.gaps()

    # The sample must be reproducible from the same seed.
    assert w.sample(25) == ObservationWindow(start, end,
                                             "CC6.1-tenant-isolation").sample(25)
    print("SOC 2 Type II evidence sampling: OK")
    print(f"  population {clean['population_days']} days, "
          f"sample {clean['sampled_days']}, opinion {clean['opinion']}")

System architecture

Tenant-Fair Inference Tier — the request path, with the decision points that must exist before the first model call:

flowchart TB
    subgraph Tenants
      TA[Tenant A - batch]
      TB[Tenant B - interactive p95 400ms]
    end
    TA --> AD{Admission}
    TB --> AD
    AD -->|bucket empty| S429[429 + Retry-After]
    AD -->|ceiling full,<br/>outside reservation| S429
    AD -->|admit| Q[Deadline queue]
    Q -->|deadline passed| SH[Shed - do not serve late]
    Q --> W[Worker pool]
    W --> P[Model / vLLM]
    P --> OBS[Per-tenant p95 / p99 window]
    OBS -->|breach| ALERT[Page + autoscale signal]

Technical parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Controls and acceptance parameters

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by 7 years for books and records (SEC 17a-4), WORM storage.

What the auditor will ask

• Show me the control-execution population for the full window, and the days where it is empty.
  Every request emits an execution record, so the population is continuous by construction. A gap is an alertable incident, not something discovered during the audit.
• Which tenants have contractual percentile targets, and what reserves capacity for them?
  Any tenant with a contractual percentile gets an entry in the reservation ledger — guaranteed in-flight slots inside the global ceiling. Reservations are validated at configuration time and must sum below the ceiling.
• What happens to tenant B's p95 when tenant A sends 10x its normal volume — measured, not assumed?
  Tenant B stays inside its SLO, because tenant A is bounded by its own token bucket and cannot consume B's reserved slots. This is testable: saturate A and watch B's p95 window.
• Is the alert on a percentile or a mean?
  A percentile. The SLA is written on p95, so the alert is on p95. The mean is dashboard-only, because it hides exactly the breach you are contractually exposed to.
• What is shed under saturation, and how does a client learn when to retry?
  Requests past their deadline are shed rather than served late, and the shed path returns 429 with an accurate Retry-After computed from the bucket's refill rate so clients back off instead of hot-looping.
• Can a cache entry created for one tenant ever be served to another?
  No. Every cache key is namespaced by tenant id; keying on prompt text alone is the mechanism by which one tenant's completion is served to another.

Observed in the field

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• Ask HN: Who wants to be hired? (November 2025) hackernews · hn/comment · 2025-11-03
• Ask HN: Who wants to be hired? (January 2026) hackernews · hn/comment · 2026-01-02
• How do you handle the CTO not being up to standard? reddit · r/ExperiencedDevs · 2026-09-25
• [Hiring] Looking for a Passionate CTO for Early-Stage Blockchain + Fintech Startup reddit · r/cto · 2024-07-07

Related reference architectures

• Part of enterprise llm architecture at scale.
• A different failure mode of the same component: EU AI Act reference architecture for enterprise LLM deployment at scale.
• This pattern also governs enterprise LLM deployment at scale.
• The same control appears in SOX reference architecture for enterprise LLM deployment at scale.
• If this is the blocker in front of you: enterprise AI infrastructure review.