Latency compliance data governance framework for enterprise LLM deployment — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · SOC 2 Type II · layout spec_first · implementation executed in CI before publish

Technical parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

The control that resolves this is a Classification-Routed Multi-Cloud Estate, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is SOC 2 Type II, and the question it actually asks is did this control operate effectively for the whole observation window. The specific problem here is latency and percentile SLAs. The shape below holds p95 ≤ 600ms over durable queue (SQS / Kafka) with an outbox, and ships with a standard-library implementation that runs as written.

Why this blocks production

Workloads span several providers and regions, and the decision of which model may see which data lives in prompt text, a wiki page, or a developer's judgement. That is not a control, so the governance board cannot approve the estate even though every individual component is defensible.

What makes latency and percentile SLAs specifically hard

The contract is written on a percentile and the dashboard shows a mean, so the breach is invisible until a customer reports it. Worse, the slowest requests are usually the most valuable ones — long context, more retrieval, more tool calls.

What resolves it is a streaming per-tenant percentile window with alerting on the percentile. The failure to design against: a p95 breach hidden under a healthy-looking average.

What SOC 2 Type II specifically requires

SOC 2 Type II is not a point-in-time assertion. The auditor samples evidence across a 3-12 month observation window and asks whether the control operated on every day of it, not whether it exists today.

The artifact that satisfies it is a sampled, time-ordered population the auditor can draw from. The way teams most often fail it: the control was added mid-window, so the population has a hole and the observation period restarts.

The implementation — Deadline propagation and percentile shedding

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind durable queue (SQS / Kafka) with an outbox; the failure mode it guards against is a consumer crash between side effect and acknowledgement. The output you hand the assessor is the observation-window evidence export.

"""Deadline propagation and percentile-aware shedding.

Why this shape: an SLA is written on a percentile, so the system has to make
per-request decisions against a deadline rather than optimise an average. Two
controls, and the second is the one teams skip:

  * every request carries a DEADLINE, and work stops when it passes -- serving
    a response nobody is waiting for costs capacity and helps no one
  * the queue is bounded by the deadline, not by a fixed length, so load sheds
    at the door instead of accumulating into a latency avalanche
"""
import time
from dataclasses import dataclass, field


class DeadlineExceeded(Exception):
    pass


@dataclass
class Deadline:
    """Monotonic, because wall-clock jumps during NTP correction and that is
    exactly when you least want the deadline logic to misbehave."""
    at: float

    @classmethod
    def in_ms(cls, ms: float) -> "Deadline":
        return cls(at=time.monotonic() + ms / 1000.0)

    def remaining_ms(self) -> float:
        return max(0.0, (self.at - time.monotonic()) * 1000.0)

    def expired(self) -> bool:
        return time.monotonic() >= self.at

    def budget_for(self, fraction: float) -> float:
        """Sub-budget for one stage, so retrieval cannot eat the whole SLA."""
        return self.remaining_ms() * fraction


class PercentileWindow:
    """Bounded streaming window. Nearest-rank, because interpolation invents
    a value that never occurred -- unhelpful when arguing about a breach."""

    def __init__(self, size: int = 4096):
        self.size, self.samples = size, []

    def observe(self, ms: float):
        self.samples.append(ms)
        if len(self.samples) > self.size:
            self.samples = self.samples[-self.size:]

    def quantile(self, q: float) -> float:
        if not self.samples:
            return 0.0
        o = sorted(self.samples)
        idx = min(len(o) - 1, max(0, int(round(q * len(o) + 0.5)) - 1))
        return o[idx]


@dataclass
class DeadlineQueue:
    """Admission bounded by whether the work can still finish in time."""
    service_time_ms: float
    concurrency: int
    inflight: int = 0
    shed: int = 0
    admitted: int = 0
    window: PercentileWindow = field(default_factory=PercentileWindow)

    def offer(self, deadline: Deadline) -> bool:
        queue_ahead = max(0, self.inflight - self.concurrency + 1)
        projected = (queue_ahead * self.service_time_ms
                     / max(self.concurrency, 1)) + self.service_time_ms
        if projected > deadline.remaining_ms():
            # Shed NOW. Accepting work that cannot meet its deadline converts
            # one late response into a queue of late responses.
            self.shed += 1
            return False
        self.inflight += 1
        self.admitted += 1
        return True

    def complete(self, elapsed_ms: float):
        self.inflight = max(0, self.inflight - 1)
        self.window.observe(elapsed_ms)

    def breaching(self, slo_ms: float, q: float = 0.95) -> bool:
        return self.window.quantile(q) > slo_ms


if __name__ == "__main__":
    q = DeadlineQueue(service_time_ms=120, concurrency=8)
    for i in range(60):
        d = Deadline.in_ms(400 if i % 3 else 150)   # mixed deadline classes
        if q.offer(d):
            q.complete(110 + (i % 7) * 9)
    print(f"admitted {q.admitted}  shed {q.shed}")
    print(f"p95 {q.window.quantile(0.95):.0f}ms  p99 {q.window.quantile(0.99):.0f}ms")
    print("breaching 400ms SLO:", q.breaching(400))

    tight = Deadline.in_ms(20)
    q.inflight = 40
    print("tight deadline under load admitted:", q.offer(tight), "(must be False)")
    print("stage budget of a 400ms deadline:",
          f"{Deadline.in_ms(400).budget_for(0.3):.0f}ms for retrieval")

The blueprint

Classification-Routed Multi-Cloud Estate — the request path, with the decision points that must exist before the first model call:

flowchart TB
    IN[Request payload] --> CLS[Deterministic classifier<br/>regex + Luhn/checksum]
    CLS --> LV{Classification}
    LV -->|PUBLIC / INTERNAL| D1[shared-us]
    LV -->|CONFIDENTIAL| D2[eu-pooled<br/>DPA required]
    LV -->|RESTRICTED| D3[eu-dedicated<br/>zero-retention + subprocessors]
    LV -->|no lawful destination| DN[Deny + reason]
    D1 --> LOG[(Decision log)]
    D2 --> LOG
    D3 --> LOG
    DN --> LOG
    D3 --> LIN[Lineage tag on<br/>embeddings / cache / evals]
    ATT[Destination attestations] -.-> LV
    ATT --> DR[Drift detector]

Implementation components

1. Deterministic classifier — Pattern plus validator (Luhn, checksum) so the gate does not cry wolf.
2. Destination attestations — Region, max class, retention, DPA, subprocessor disclosure — as data.
3. Routing gate — First lawful destination in preference order; cost can never override a control.
4. Decision log — A recorded reason for every allow AND every deny.
5. Lineage tags — Classification travels with derived artifacts: embeddings, caches, evals.
6. Drift detector — Alerts when a destination's attested properties change.

Risk and control matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Review questions and answers

• Show me the control-execution population for the full window, and the days where it is empty.
  Every request emits an execution record, so the population is continuous by construction. A gap is an alertable incident, not something discovered during the audit.
• Is your alert on a percentile or a mean, and which percentile is in the contract?
  The alert is on the contracted percentile, computed per tenant over a bounded streaming window. The mean is displayed but never alertable, because it hides exactly the breach we are exposed to.
• What function decides which destination may see RESTRICTED data, and where does it run?
  The routing gate, evaluated before egress. It is a deterministic function of the data's classification and the destination's attested properties, and it fails closed when no destination is lawful.
• Show me the recorded reason for a denial last week.
  Every allow and every deny is written to the decision log with the reason — which destinations were rejected and on which property — so a denial is explainable months later.
• When a document is embedded, does the embedding carry its classification — and is that enforced on read?
  Yes. The classification tag propagates into every derived store, including embeddings, caches and eval sets, and is enforced on read as well as write.
• What happens when a provider adds a subprocessor you have not disclosed?
  The drift detector alerts when a destination's attested properties change, and routing is re-evaluated against the new attestation. Attestations are versioned data, not a one-off questionnaire.
• How do you prove processing stayed in the contracted region?
  The provider endpoint is pinned per region and the served region is asserted on the response; a mismatch fails the request rather than completing it quietly.

Observed in the field

This page exists because the problem is reported publicly. The threads below are the primary sources, linked unmodified and not affiliated with this site.

• feat(callbacks): In-memory Zero-Trust Data Sanitization (ZTDS) callback (IETF draft-02) github · langchain-ai/langchain · 2026-09-27

Related reference architectures

• Part of enterprise ai data governance.
• This pattern also governs data residency data governance framework for enterprise LLM deployment.
• For the related constraint, see HIPAA data governance framework for enterprise LLM deployment.
• If this is the blocker in front of you: enterprise AI infrastructure review.