Multi-tenant architecture for enterprise LLM deployment at scale — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · SOC 2 Type II · layout diagram_first · implementation executed in CI before publish

The control that resolves this is a Tenant-Fair Inference Tier, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The specific problem here is tenant isolation. The shape below holds p95 ≤ 450ms over REST over HTTPS, and ships with a standard-library implementation that runs as written.

The request path

Tenant-Fair Inference Tier — the request path, with the decision points that must exist before the first model call:

flowchart TB
    subgraph Tenants
      TA[Tenant A - batch]
      TB[Tenant B - interactive p95 400ms]
    end
    TA --> AD{Admission}
    TB --> AD
    AD -->|bucket empty| S429[429 + Retry-After]
    AD -->|ceiling full,<br/>outside reservation| S429
    AD -->|admit| Q[Deadline queue]
    Q -->|deadline passed| SH[Shed - do not serve late]
    Q --> W[Worker pool]
    W --> P[Model / vLLM]
    P --> OBS[Per-tenant p95 / p99 window]
    OBS -->|breach| ALERT[Page + autoscale signal]
Request path. Rendered as a diagram where scripting is available; the Mermaid source above is the authoritative definition either way.

Component breakdown

  1. Per-tenant token bucket — Sustained rate plus burst, so one tenant cannot spend another's budget.
  2. Reservation ledger — Guaranteed in-flight slots per tenant; reservations must sum below the ceiling.
  3. Global concurrency ceiling — Bounds the shared tier so it degrades predictably instead of collapsing.
  4. Percentile window — Streaming p95/p99 per tenant — the number the contract is written on.
  5. Queue with deadline — Requests carry a deadline and are shed rather than served late.
  6. Shed path — 429 with an accurate Retry-After so clients back off instead of hot-looping.

Implementation parameter matrix

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Implementation parameters for this deployment shape
Deployment vectorDedicated inference tier with per-tenant admission control
Security standardSOC 2 Type II
Applicable regimesSOC 2 Type II, ISO/IEC 27001:2022, GDPR
Latency SLAp95 ≤ 450ms — interactive surfaces need sub-second; batch jobs do not
Integration protocolREST over HTTPS — the widest compatibility and the easiest audit story
Idempotency controlan Idempotency-Key header, honoured for 24h
Throughput envelope~2k rps per gateway node before connection churn dominates
Middleware requirementpolicy decision point in front of the model gateway
Data classes in scopePII, commercial-in-confidence data
Retention obligationper the data class retention schedule, enforced automatically
Data residencycontracted processing region, asserted on every response
Contractual instrumentDPA plus a disclosed subprocessor list
Systems of recordSAP, Salesforce, Workday
Primary failure modea retried POST creating a duplicate record

The control gap

A shared inference tier holds under average load and breaches its p95 under a neighbour's batch job. The SLA is written on a percentile, the dashboards show a mean, and admission control is a single global rate limit that protects the provider rather than any individual tenant.

What makes tenant isolation specifically hard

Tenancy is a correctness property before it is a performance one. The question is not whether tenants share hardware, it is whether any code path can serve one tenant's data to another — through a cache key, a vector index, a log, or a prompt built from the wrong context.

What resolves it is a tenant id that is structural in every key, index and record. The failure to design against: a cache keyed on prompt text alone, serving tenant A's completion to tenant B.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

Why this blocks a rollout rather than slowing it: Noisy-neighbour latency cannot be fixed downstream of admission. By the time a request is queued, the only remaining options are to shed it or to breach someone's contract. In regulated enterprise the cost of getting it wrong is concrete — a failed Type II observation window restarts the clock.

Implementation artifact — Per-tenant admission control and percentile tracking

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind REST over HTTPS; the failure mode it guards against is a retried POST creating a duplicate record. The output you hand the assessor is the observation-window evidence export.

"""Per-tenant admission control for a shared inference tier.

Why this shape: a global rate limit protects the provider and nobody else. The
failure enterprises actually hit is the noisy-neighbour one — tenant A's batch
job saturates the queue and tenant B breaches the p95 latency SLA that is
written into their contract. You cannot fix that downstream of admission.

Two controls, both required:
  * a per-tenant token bucket, so one tenant cannot consume another's budget
  * a global concurrency ceiling with per-tenant reservations, so the shared
    tier degrades predictably instead of collapsing
"""
import heapq
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    rate: float            # sustained requests per second
    burst: float           # instantaneous allowance
    tokens: float = None
    updated: float = field(default_factory=time.monotonic)

    def __post_init__(self):
        if self.rate <= 0 or self.burst <= 0:
            raise ValueError("rate and burst must be positive")
        if self.tokens is None:
            self.tokens = self.burst

    def take(self, n: float = 1.0, now: float = None) -> bool:
        now = time.monotonic() if now is None else now
        elapsed = max(0.0, now - self.updated)
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
        self.updated = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

    def retry_after(self, n: float = 1.0) -> float:
        """Seconds until n tokens exist. Return this in a 429 Retry-After so
        clients back off correctly instead of hot-looping."""
        deficit = max(0.0, n - self.tokens)
        return deficit / self.rate


class Admission:
    """Per-tenant buckets plus a global concurrency ceiling with reservations.

    reserved: tenant -> guaranteed in-flight slots. The sum of reservations
    must stay below `ceiling`, or the guarantee is fiction.
    """

    def __init__(self, ceiling: int, reserved: dict = None):
        self.ceiling = ceiling
        self.reserved = dict(reserved or {})
        if sum(self.reserved.values()) > ceiling:
            raise ValueError(
                f"reservations {sum(self.reserved.values())} exceed ceiling "
                f"{ceiling}: at least one tenant's SLA cannot be honoured")
        self.buckets: dict = {}
        self.inflight: dict = {}

    def configure(self, tenant: str, rate: float, burst: float):
        self.buckets[tenant] = TokenBucket(rate=rate, burst=burst)

    def admit(self, tenant: str, cost: float = 1.0) -> tuple:
        """-> (admitted, reason, retry_after_seconds)"""
        bucket = self.buckets.get(tenant)
        if bucket is None:
            return False, "tenant_not_provisioned", None

        used = sum(self.inflight.values())
        mine = self.inflight.get(tenant, 0)
        my_reservation = self.reserved.get(tenant, 0)

        # A tenant inside its reservation is admitted even when the shared pool
        # is full — that is what makes the reservation worth signing.
        if used >= self.ceiling and mine >= my_reservation:
            return False, "global_ceiling", 0.05

        if not bucket.take(cost):
            return False, "tenant_rate_limit", round(bucket.retry_after(cost), 3)

        self.inflight[tenant] = mine + 1
        return True, "ok", None

    def release(self, tenant: str):
        self.inflight[tenant] = max(0, self.inflight.get(tenant, 0) - 1)


class LatencyWindow:
    """Streaming p95/p99 over a bounded window.

    An average hides exactly the breach you are contractually exposed to: an
    SLA is written on a percentile, so the number you alert on must be one.
    """

    def __init__(self, size: int = 2048):
        self.size = size
        self.samples: list = []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        ordered = sorted(self.samples)
        # Nearest-rank: the smallest sample at or above the q-th position.
        idx = min(len(ordered) - 1, max(0, int(round(q * len(ordered) + 0.5)) - 1))
        return ordered[idx]

    def breach(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.quantile(q) > slo_ms


if __name__ == "__main__":
    adm = Admission(ceiling=10, reserved={"bank-eu": 4})
    adm.configure("bank-eu", rate=5, burst=5)
    adm.configure("retail-us", rate=50, burst=50)
    for _ in range(9):
        adm.admit("retail-us")
    print("noisy neighbour holds", adm.inflight)
    print("reserved tenant still admitted:", adm.admit("bank-eu"))
    print("burst exhausted ->", [adm.admit("bank-eu") for _ in range(5)][-1])

    win = LatencyWindow()
    for ms in [120, 140, 135, 128, 900, 131, 139, 142, 150, 2200]:
        win.observe(ms)
    print(f"p95={win.quantile(0.95):.0f}ms breach(400ms)={win.breach(400)}")

Risk mitigation matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

CategoryRiskControlAcceptance parameter
SOC 2 Type IIEvidence population has gaps across the observation windowEmit a control-execution record on every request, not on exceptions; alert when daily volume drops below the expected floor0 days with no control records across the window
tenant isolationCross-tenant leakage through a shared cache or vector indexNamespace every cache key, index partition and log record by tenant id; make the unnamespaced call impossible in the clientCache key audit shows 100% tenant-namespaced; cross-tenant hit rate 0 under an adversarial test
Latency SLAOne tenant's batch traffic breaches another's contractual p95Per-tenant token buckets plus reserved in-flight slots inside a global ceilingReserved tenants hold p95 under their SLO while a neighbour saturates the tier
Latency SLAMean latency hides the percentile the contract is written onTrack streaming p95/p99 per tenant; alert on the percentile, never the averageAlert fires on p95 breach within one window; mean is dashboard-only
Latency SLALate responses served instead of shed, so queues grow unboundedlyAttach a deadline to every request and shed on expiry0 responses served after deadline; shed rate published per tenant
Data privacyCross-tenant leakage through a shared cache or KV reuseNamespace every cache key by tenant id; never key on prompt text aloneCache key audit shows 100% tenant-namespaced; cross-tenant hit rate 0
Cost controlOver-provisioning GPU to absorb one tenant's burstsBound burst at the bucket, scale on admitted load rather than offered loadGPU utilisation 60-80%; scaling driven by admitted load
Cost controlRetries amplify a degradation into an outageAccurate Retry-After, client backoff, and a circuit breaker on the shed pathRetry amplification under 1.3x during an induced degradation
OperationalReservations sum above the ceiling, so guarantees are fictionValidate at configuration time; refuse to start with over-committed reservationsStartup check rejects over-commitment; covered by a unit test

Questions to answer before approval

Observed in the field

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

Related architectures

Enterprise infrastructure review

If you are carrying this blocker, the useful next step is not a demo. It is a review of your actual request path against the control points above:

  • Which of the 6 components exist in your estate today, and which are assumed
  • Your current numbers against the 7 acceptance parameters, starting with p95 ≤ 450ms
  • Which SOC 2 Type II control questions you cannot yet answer, and what closes each gap
Request an architecture review