HIPAA vendor evaluation criteria for enterprise AI implementation — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · HIPAA · layout control_review · implementation executed in CI before publish

The control that resolves this is a Gated Vendor Evaluation Framework, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is HIPAA, and the question it actually asks is can PHI reach a place the BAA does not cover. The specific problem here is RFP and structured evaluation. The shape below holds p95 ≤ 600ms over outbound webhooks with signed payloads, and ships with a standard-library implementation that runs as written.

The architecture friction point

A vendor comparison built as a weighted average lets a supplier average its way past a control it does not have. The scorecard ranks everyone, the committee cannot see which gates were failed, and the decision gets re-litigated after the contract is signed.

What makes RFP and structured evaluation specifically hard

An RFP scored as a weighted average lets a vendor average its way past a control it does not have. The scoring model, not the questions, is usually what makes the evaluation indefensible.

What resolves it is a two-layer scorecard: pass/fail gates, then weights. The failure to design against: the highest-scoring vendor failing security review after award.

What HIPAA specifically requires

HIPAA holds the covered entity responsible for PHI wherever it flows. A model provider without a signed BAA is not a permissible destination, and 'the prompt only contained a little PHI' is not a defence — the minimum necessary standard is itself a control.

The artifact that satisfies it is a de-identification record per request plus the BAA chain. The way teams most often fail it: PHI reaching a subprocessor with no BAA, via a logging sidecar nobody mapped.

Risk register with acceptance criteria

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Implementation parameter matrix

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

What the auditor will ask

• Which destinations hold a BAA, and what structurally prevents PHI from reaching one that does not?
  Destinations carry a BAA flag as attested data, and the routing gate refuses a PHI-classified payload to any destination without it. The refusal is logged with its reason.
• Which criteria are pass/fail, and which vendor failed one?
  Gates are evaluated first and independently of score: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, residency. A failure on any of them disqualifies.
• Which criteria are pass/fail, and which vendor failed one?
  The gate layer is pass/fail: SOC 2 Type 2 currency, DPA and subprocessor disclosure, tenant isolation evidence, and data residency. Gates are evaluated before any weighted scoring, so a vendor cannot average past a control it lacks.
• For each passed gate, what artifact proves it and what is its date?
  Each gate names the artifact that satisfies it — the report, the signed DPA, the architecture document — with its date. A claim is not evidence.
• Is the latency target in the signed agreement with a remedy, or on a status page?
  It must be in the signed agreement with a remedy. A status page is not a commitment; the scorecard scores the remedy, not the promise.
• What does exit look like — specifically, what can you export and has it been rehearsed?
  Export of prompts, embeddings, eval sets and fine-tunes, rehearsed during the pilot rather than promised in the contract. Portability that has never been executed is not portability.
• Was the reference deployment in the same vertical and regulatory regime as ours?
  It must be — a reference in the same vertical and the same regulatory regime, worked through a structured question set, not a vendor-selected happy customer.

System architecture

Gated Vendor Evaluation Framework — the request path, with the decision points that must exist before the first model call:

flowchart TB
    V[Vendor responses] --> G{Gates - pass/fail}
    G -->|any gate failed| DQ[Disqualified<br/>with named gates]
    G -->|all gates passed| W[Weighted scoring]
    W --> TCO[3-year fully loaded TCO]
    W --> REF[Reference call<br/>comparable regulated deployment]
    TCO --> R[Ranking]
    REF --> R
    R --> DR[Decision record<br/>score + reason per criterion]
    DQ --> DR
    DR --> SC[Steering committee]

Modular implementation framework

1. Gate layer — Pass/fail controls: SOC 2 Type 2 currency, DPA, tenant isolation, residency.
2. Weighted layer — Ranks only vendors that cleared every gate.
3. Evidence requirement — Each gate names the artifact that satisfies it, not a claim.
4. TCO model — Fully loaded three-year: licence, run cost, internal effort.
5. Reference protocol — A structured call with a comparable regulated deployment.
6. Decision record — Ranking plus failed gates plus a written reason per score.

The implementation — Weighted scorecard with disqualifying gates

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind outbound webhooks with signed payloads; the failure mode it guards against is at-least-once delivery handled as exactly-once by the consumer. The output you hand the assessor is the minimum-necessary de-identification log.

"""Weighted vendor scorecard with disqualifying gates.

Why this shape: a procurement scorecard that is a weighted average lets a vendor
average its way past a control it simply does not have. Real evaluations have
two layers — GATES that are pass/fail, and weights that only ever rank the
vendors that already cleared every gate.

This produces the artifact a steering committee signs: a ranking, the gates each
vendor failed, and a written reason per score.
"""
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Criterion:
    key: str
    weight: float          # relative; normalised at scoring time
    gate: bool = False     # True -> pass/fail, cannot be averaged away
    note: str = ""


DEFAULT_CRITERIA = [
    Criterion("soc2_type2_current", 0.0, gate=True,
              note="Type 2 report inside 12 months, not a Type 1 or a roadmap"),
    Criterion("dpa_and_subprocessors", 0.0, gate=True,
              note="DPA signable as-is and a disclosed subprocessor list"),
    Criterion("tenant_isolation_evidence", 0.0, gate=True,
              note="Architectural isolation, not a WHERE clause"),
    Criterion("data_residency", 0.0, gate=True,
              note="Processing stays in the contracted region"),
    Criterion("exit_and_portability", 2.0,
              note="Export of prompts, embeddings, evals; no model lock-in"),
    Criterion("latency_slo_contractual", 3.0,
              note="p95 in the agreement with a remedy, not a status page"),
    Criterion("integration_depth", 3.0,
              note="Works against the real system of record, not a demo schema"),
    Criterion("implementation_capacity", 2.0,
              note="Named engineers, not a bench claim"),
    Criterion("tco_3yr", 3.0, note="Fully loaded: licence, run, internal effort"),
    Criterion("reference_in_our_vertical", 2.0,
              note="A reference call with a comparable regulated deployment"),
]


class Disqualified(Exception):
    pass


@dataclass
class Vendor:
    name: str
    gates: dict = field(default_factory=dict)    # key -> bool
    scores: dict = field(default_factory=dict)   # key -> 0..5
    reasons: dict = field(default_factory=dict)


def evaluate(vendors, criteria=None):
    """-> (ranked, disqualified). Gates are evaluated first, always."""
    criteria = criteria or DEFAULT_CRITERIA
    gate_keys = [c.key for c in criteria if c.gate]
    weighted = [c for c in criteria if not c.gate and c.weight > 0]
    total_w = sum(c.weight for c in weighted) or 1.0

    ranked, disqualified = [], []
    for v in vendors:
        failed = [k for k in gate_keys if not v.gates.get(k, False)]
        if failed:
            disqualified.append({"vendor": v.name, "failed_gates": failed})
            continue
        got = 0.0
        breakdown = []
        for c in weighted:
            s = float(v.scores.get(c.key, 0))
            if not 0 <= s <= 5:
                raise ValueError(f"{v.name}.{c.key}={s} outside 0..5")
            contribution = (c.weight / total_w) * (s / 5.0)
            got += contribution
            breakdown.append({"criterion": c.key, "score": s,
                              "weight": c.weight,
                              "contribution": round(contribution, 4),
                              "reason": v.reasons.get(c.key, ""),
                              "standard": c.note})
        ranked.append({"vendor": v.name, "score": round(got * 100, 1),
                       "breakdown": breakdown})
    ranked.sort(key=lambda r: -r["score"])
    return ranked, disqualified


if __name__ == "__main__":
    passing = {k: True for k in
               ("soc2_type2_current", "dpa_and_subprocessors",
                "tenant_isolation_evidence", "data_residency")}
    a = Vendor("Vendor A", gates=dict(passing),
               scores={"exit_and_portability": 4, "latency_slo_contractual": 5,
                       "integration_depth": 4, "implementation_capacity": 3,
                       "tco_3yr": 3, "reference_in_our_vertical": 5},
               reasons={"latency_slo_contractual": "p95 450ms with 10% credit"})
    b = Vendor("Vendor B", gates={**passing, "tenant_isolation_evidence": False},
               scores={k: 5 for k in
                       ("exit_and_portability", "latency_slo_contractual",
                        "integration_depth", "implementation_capacity",
                        "tco_3yr", "reference_in_our_vertical")})
    ranked, dq = evaluate([a, b])
    for r in ranked:
        print(f"{r['vendor']}: {r['score']}")
    print("disqualified despite perfect scores:", dq)

The HIPAA control, as code

Minimum-necessary de-identification with a BAA-pinned egress gate. This is the artifact an assessor asks for under HIPAA specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Minimum-necessary de-identification with a BAA-pinned destination gate.

HIPAA holds the covered entity responsible for PHI wherever it flows, so the
control is not "we told people not to paste PHI". Two things must be
structural: PHI is reduced to the minimum necessary BEFORE egress, and the
destination set is pinned to those under a signed BAA, failing closed on
anything else.

Safe Harbor (45 CFR 164.514(b)(2)) lists 18 identifier classes. This implements
the mechanical subset and records a per-request de-identification receipt,
because the auditor asks for proof per request, not a policy document.
"""
import re

SAFE_HARBOR_PATTERNS = {
    "name_mrn": re.compile(r"\bMRN[:\s#]*([A-Z0-9-]{4,})", re.I),
    "ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
    "phone": re.compile(r"\b(?:\+1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b"),
    "email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b"),
    "date_full": re.compile(r"\b\d{1,2}/\d{1,2}/(19|20)\d{2}\b"),
    "member_id": re.compile(r"\b(?:member|policy)\s*(?:id|#)[:\s]*([A-Z0-9-]{5,})", re.I),
}


class BAARegistry:
    """Destinations are permissible only as attested data, never by convention."""

    def __init__(self):
        self._signed: dict[str, str] = {}

    def attest(self, destination: str, agreement_ref: str) -> None:
        self._signed[destination] = agreement_ref

    def covered(self, destination: str) -> bool:
        return destination in self._signed

    def reference(self, destination: str) -> str | None:
        return self._signed.get(destination)


class PHIEgressGate:
    def __init__(self, registry: BAARegistry):
        self.registry = registry
        self.receipts: list[dict] = []

    def deidentify(self, text: str) -> tuple[str, dict[str, int]]:
        counts: dict[str, int] = {}
        out = text
        for label, pattern in SAFE_HARBOR_PATTERNS.items():
            out, n = pattern.subn(f"[REDACTED:{label}]", out)
            if n:
                counts[label] = n
        return out, counts

    def send(self, text: str, destination: str) -> dict:
        """Refuses rather than redacting-and-hoping when the BAA is absent."""
        redacted, counts = self.deidentify(text)
        phi_present = bool(counts)
        if phi_present and not self.registry.covered(destination):
            receipt = {"destination": destination, "admitted": False,
                       "reason": "no BAA on file for a PHI-classified payload",
                       "identifier_classes": sorted(counts)}
            self.receipts.append(receipt)
            raise PermissionError(receipt["reason"])
        receipt = {
            "destination": destination,
            "admitted": True,
            "baa_ref": self.registry.reference(destination),
            "identifier_classes": sorted(counts),
            "redactions": sum(counts.values()),
            "payload": redacted,
        }
        self.receipts.append(receipt)
        return receipt


if __name__ == "__main__":
    reg = BAARegistry()
    reg.attest("azure-openai-us-east", "BAA-2026-0041")
    gate = PHIEgressGate(reg)

    note = ("Patient MRN: 88213A, contact 555-867-5309 or a.patient@example.com, "
            "DOB 04/11/1974, follow-up scheduled.")
    ok = gate.send(note, "azure-openai-us-east")
    assert ok["admitted"] and ok["redactions"] >= 4, ok
    assert "88213A" not in ok["payload"] and "555-867-5309" not in ok["payload"], ok
    assert "a.patient@example.com" not in ok["payload"], ok

    try:
        gate.send(note, "some-vendor-eval-endpoint")
        raise AssertionError("PHI reached a destination with no BAA")
    except PermissionError:
        pass

    # No PHI means no BAA requirement — the gate must not over-block.
    benign = gate.send("Summarise ward occupancy trends.", "some-vendor-eval-endpoint")
    assert benign["admitted"] and benign["redactions"] == 0, benign

    print("HIPAA minimum-necessary egress gate: OK")
    print(f"  receipts {len(gate.receipts)}, "
          f"refusals {sum(1 for r in gate.receipts if not r['admitted'])}")

Related architectures

• Part of enterprise ai vendor evaluation.
• Where this interacts with another regime, see GDPR vendor evaluation criteria for enterprise AI implementation.
• This pattern also governs EU AI Act vendor evaluation criteria for enterprise AI implementation.
• This pattern also governs ISO/IEC 27001:2022 vendor evaluation criteria for enterprise AI.
• SOX vendor evaluation criteria for enterprise AI implementation applies the same mechanism under a different obligation.
• If this is the blocker in front of you: enterprise AI infrastructure review.