GDPR legacy integration patterns for enterprise AI agents — Reference Architecture

Reference architecture · published 2026-10-16 · regulated enterprise · GDPR · layout walkthrough · implementation executed in CI before publish

The control that resolves this is an Idempotent System-of-Record Bridge, evaluated before the request reaches a model provider rather than audited afterwards. In regulated enterprise the binding constraint is GDPR, and the question it actually asks is what is the lawful basis, and can you actually erase. The specific problem here is mainframe integration. The shape below holds p95 ≤ 2400ms over durable queue (SQS / Kafka) with an outbox, and ships with a standard-library implementation that runs as written.

Where the rollout actually stalls

Agents can read the legacy estate and cannot safely write to it. A retry after a gateway timeout creates a duplicate posting, and in an ERP a duplicate journal entry is a finance incident. So the team that owns the system of record withholds write access, and the automation stalls at read-only.

What makes mainframe integration specifically hard

The mainframe is the system of record and its change window is measured in quarters. Anything that requires a change on that side is not an integration plan, it is a request to join a release queue.

What resolves it is a side-car integration that requires zero mainframe change. The failure to design against: a design whose first step is a CICS change nobody will schedule.

What GDPR specifically requires

GDPR asks two things an audit log does not answer: under what lawful basis was this processed, and can you erase it on request. Erasure is the harder one — a prompt embedded into a vector index and a fine-tune has propagated beyond the row you deleted.

The artifact that satisfies it is a lineage graph from source record to every derived artifact. The way teams most often fail it: an erasure request that cannot be honoured because the data survives in embeddings, caches and eval sets.

The parts you have to build

1. Content-derived idempotency key — Derived from business keys, so two attempts at one intent collide.
2. Claim table — Transactional claim in the same database as the outbox; a retry observes the first attempt.
3. Typed write contract — Allow-listed fields per document type; unknown fields are rejected loudly.
4. Outbox — Durable intent record, so a crash mid-flight is recoverable rather than ambiguous.
5. Reconciliation job — Compares posted documents against intents and surfaces drift.
6. Compensating action — A defined reversal path per document type, because some writes must be undoable.

Reference implementation — Idempotent write-back with a typed write contract

Standard-library Python, no external dependencies, runs as written. Executed by the publish gate: if it raises, this page does not ship. Wire it behind durable queue (SQS / Kafka) with an outbox; the failure mode it guards against is a consumer crash between side effect and acknowledgement. The output you hand the assessor is the data-subject lineage and erasure report.

"""Idempotent write-back adapter for a legacy system of record (SAP/ERP shape).

Why this shape: the blocker on legacy integration is almost never reading. It
is writing. An agent that retries a posting because a gateway timed out will
happily create a duplicate journal entry, and in an ERP a duplicate posting is
a finance incident, not a bug report.

Three controls make agent-driven writes acceptable to the team that owns the
system of record:
  * a deterministic idempotency key derived from business content
  * a claim table, so a retry observes the first attempt instead of repeating it
  * a bounded, typed projection, so the agent can never post a field it does
    not understand
"""
import hashlib
import json
import time
from dataclasses import dataclass


class DuplicatePosting(Exception):
    """Raised when a retry would create a second document for one intent."""


def idempotency_key(doc_type: str, business_keys: dict) -> str:
    """Derived from BUSINESS content, never from a uuid4.

    A random key makes every retry unique, which is the opposite of what
    idempotency means. Two attempts to post the same invoice must collide.
    """
    canonical = json.dumps({k: business_keys[k] for k in sorted(business_keys)},
                           sort_keys=True, separators=(",", ":"))
    return f"{doc_type}:" + hashlib.sha256(canonical.encode()).hexdigest()[:32]


ALLOWED_FIELDS = {
    "journal_entry": {"company_code", "posting_date", "currency",
                      "debit_account", "credit_account", "amount", "reference"},
}


def project(doc_type: str, payload: dict) -> dict:
    """Allow-list projection. An agent-proposed field that is not in the
    contract is dropped loudly, not posted hopefully."""
    allowed = ALLOWED_FIELDS.get(doc_type)
    if allowed is None:
        raise ValueError(f"no write contract defined for {doc_type!r}")
    unknown = set(payload) - allowed
    if unknown:
        raise ValueError(f"fields outside the write contract: {sorted(unknown)}")
    missing = allowed - set(payload)
    if missing:
        raise ValueError(f"write contract requires: {sorted(missing)}")
    return {k: payload[k] for k in sorted(allowed)}


@dataclass
class ClaimStore:
    """Stands in for a real transactional table. In production this is a row
    in the SAME database as your outbox, claimed with SELECT ... FOR UPDATE."""

    def __post_init__(self):
        self._claims: dict = {}

    def claim(self, key: str, ttl: float = 900.0) -> tuple:
        """-> (is_new, existing_result). Atomic in a real DB."""
        now = time.time()
        rec = self._claims.get(key)
        if rec and (rec["status"] == "done" or now - rec["at"] < ttl):
            return False, rec
        self._claims[key] = {"status": "in_flight", "at": now, "result": None}
        return True, None

    def complete(self, key: str, result: dict):
        self._claims[key] = {"status": "done", "at": time.time(), "result": result}

    def fail(self, key: str):
        self._claims.pop(key, None)      # let a genuine retry through


def post_document(store: ClaimStore, doc_type: str, business_keys: dict,
                  payload: dict, transport) -> dict:
    """The only sanctioned path from an agent to the system of record."""
    key = idempotency_key(doc_type, business_keys)
    body = project(doc_type, payload)

    is_new, existing = store.claim(key)
    if not is_new:
        if existing["status"] == "done":
            # The correct response to a duplicate is the ORIGINAL result.
            return {"idempotency_key": key, "replayed": True,
                    **existing["result"]}
        raise DuplicatePosting(f"{key} is already in flight")

    try:
        result = transport(doc_type, body, key)
    except Exception:
        store.fail(key)
        raise
    store.complete(key, result)
    return {"idempotency_key": key, "replayed": False, **result}


if __name__ == "__main__":
    calls = []

    def fake_sap(doc_type, body, key):
        calls.append(key)
        return {"document_number": f"50000{len(calls)}", "posted": True}

    store = ClaimStore()
    keys = {"invoice_id": "INV-8831", "company_code": "1000"}
    payload = {"company_code": "1000", "posting_date": "2026-10-16",
               "currency": "EUR", "debit_account": "400000",
               "credit_account": "113100", "amount": "14500.00",
               "reference": "INV-8831"}

    first = post_document(store, "journal_entry", keys, payload, fake_sap)
    retry = post_document(store, "journal_entry", keys, payload, fake_sap)
    print("first :", first["document_number"], "replayed", first["replayed"])
    print("retry :", retry["document_number"], "replayed", retry["replayed"])
    print("transport invocations:", len(calls), "(must be 1)")
    try:
        project("journal_entry", {**payload, "tax_code": "V1"})
    except ValueError as e:
        print("contract rejects unknown field:", e)

The GDPR control, as code

Article 17 erasure propagated into derived artifacts. This is the artifact an assessor asks for under GDPR specifically — it is not the same control as the one above, and a system that implements one does not thereby implement the other. It executes under the same publish gate.

"""Article 17 erasure that propagates into derived artifacts.

The erasure request everyone implements is a DELETE on the source row. The one
that fails an audit is the derived copy: embeddings in a vector index, a cached
retrieval chunk, a fine-tuning corpus, a prompt log. Article 17(2) expects
controllers to take reasonable steps to inform processors of the erasure, and a
vector built from personal data is still personal data.

So erasure is modelled as a lineage walk with a receipt per derived artifact,
and the request is not complete until every downstream store confirms.
"""
from datetime import datetime, timezone


class Store:
    def __init__(self, name: str, kind: str):
        self.name, self.kind = name, kind
        self.rows: dict[str, dict] = {}
        self.erased: list[str] = []

    def put(self, key: str, subject_id: str, payload: str) -> None:
        self.rows[key] = {"subject_id": subject_id, "payload": payload}

    def erase_subject(self, subject_id: str) -> int:
        doomed = [k for k, v in self.rows.items() if v["subject_id"] == subject_id]
        for k in doomed:
            del self.rows[k]
            self.erased.append(k)
        return len(doomed)

    def holds(self, subject_id: str) -> int:
        return sum(1 for v in self.rows.values() if v["subject_id"] == subject_id)


class ErasureOrchestrator:
    """Lineage-aware erasure. A store that is not registered cannot be erased,
    so registration is the control — an unregistered derived store is exactly
    how personal data survives a 'completed' erasure."""

    def __init__(self):
        self.stores: list[Store] = []
        self.lineage: dict[str, list[str]] = {}

    def register(self, store: Store, derived_from: str | None = None) -> Store:
        self.stores.append(store)
        if derived_from:
            self.lineage.setdefault(derived_from, []).append(store.name)
        return store

    def execute(self, subject_id: str) -> dict:
        receipts = []
        for store in self.stores:
            held = store.holds(subject_id)
            removed = store.erase_subject(subject_id)
            receipts.append({"store": store.name, "kind": store.kind,
                             "held": held, "erased": removed})
        residual = {s.name: s.holds(subject_id) for s in self.stores
                    if s.holds(subject_id)}
        return {
            "subject_id": subject_id,
            "completed_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "receipts": receipts,
            "residual": residual,
            "complete": not residual,
            "derived_stores_covered": sorted(
                s.name for s in self.stores if s.kind != "source"),
        }


if __name__ == "__main__":
    orch = ErasureOrchestrator()
    src = orch.register(Store("crm_postgres", "source"))
    vec = orch.register(Store("vector_index", "embedding"), derived_from="crm_postgres")
    cache = orch.register(Store("retrieval_cache", "cache"), derived_from="vector_index")
    logs = orch.register(Store("prompt_log", "log"), derived_from="crm_postgres")

    src.put("c-1", "subject-77", "Jana Soor, contract renewal notes")
    vec.put("v-1", "subject-77", "[0.11, -0.4, ...]")
    cache.put("k-1", "subject-77", "chunk: contract renewal notes")
    logs.put("l-1", "subject-77", "prompt containing the same notes")
    src.put("c-2", "subject-90", "unrelated subject")

    result = orch.execute("subject-77")
    assert result["complete"], result["residual"]
    assert len(result["derived_stores_covered"]) == 3, result
    assert sum(r["erased"] for r in result["receipts"]) == 4, result
    assert src.holds("subject-90") == 1, "erasure must not over-reach"

    # An unregistered derived store is the failure this design exists to expose.
    orphan = Store("analytics_extract", "warehouse")
    orphan.put("a-1", "subject-77", "same personal data, different copy")
    assert orphan.holds("subject-77") == 1
    assert orphan.name not in {s.name for s in orch.stores}

    print("GDPR Article 17 lineage erasure: OK")
    print(f"  stores erased {len(result['receipts'])}, complete={result['complete']}")
    print(f"  UNREGISTERED store still holding data: {orphan.name} "
          f"({orphan.holds('subject-77')} row) — registration is the control")

System architecture

Idempotent System-of-Record Bridge — the request path, with the decision points that must exist before the first model call:

flowchart LR
    AG[Agent] --> PR[Projection<br/>allow-listed fields]
    PR -->|unknown field| REJ[Reject - loud]
    PR --> IK[Idempotency key<br/>sha256 of business keys]
    IK --> CL{Claim table}
    CL -->|already done| RP[Replay original result]
    CL -->|in flight| DUP[Refuse duplicate]
    CL -->|new| OB[(Outbox)]
    OB --> TR[Transport - OData / IDoc / RFC]
    TR --> SOR[(System of record<br/>SAP / ERP / AS400)]
    SOR --> RC[Reconciliation]
    RC -->|drift| CMP[Compensating action]

Enterprise implementation parameters

Every value below is a parameter you can hold a vendor or an internal team to. Adjectives are not acceptance criteria.

Risk and control matrix

Each row carries an acceptance parameter rather than an adjective, because "low latency" cannot be signed off. Retention here is bound by per the data class retention schedule, enforced automatically.

Review questions and answers

• A data subject requests erasure. Trace exactly what gets deleted, including embeddings and cached completions.
  Every derived artifact carries the source record id, so erasure traverses the lineage graph. The completion report lists each store touched and the rows removed.
• What does this require the mainframe team to change, and when is their next window?
  Nothing. All new behaviour lives in the side-car and the existing interface is consumed as-is, so delivery does not depend on their release queue.
• If the gateway times out and the agent retries, how many documents exist — and what proves it?
  One. The idempotency key is derived from the business keys, so both attempts collide on the same claim; the retry observes the completed claim and replays the original result. The transport is invoked once, which the test asserts.
• Which exact fields may an agent write per document type, and what happens to a field outside that list?
  Only the fields in that document type's allow-list, and all of them — the projection rejects both unknown fields and incomplete payloads. A field outside the contract raises rather than posting hopefully.
• What does the system-of-record owner see when a posting is wrong, and what reverses it?
  The reconciliation job compares posted documents against recorded intents and surfaces drift; each writable document type has a defined, rehearsed compensating action for reversal.
• Where is the intent recorded before the transport call, and what resolves a mid-flight crash?
  In the outbox, written before the transport call. On restart, reconciliation resolves anything left in flight, so a crash leaves a recoverable state rather than an ambiguous one.
• Is the extract pulling the minimum field set, with a justification per field?
  It should be — projection applies on read as well as write, with a recorded justification per field, so the extract schema matches the documented minimum field set.

Related architectures

• Part of legacy system integration with ai agents.
• SOC 2 Type II legacy integration patterns for enterprise AI agents applies the same mechanism under a different obligation.
• This pattern also governs ISO/IEC 27001:2022 legacy integration patterns for enterprise AI agents.
• Where this interacts with another regime, see HIPAA legacy integration patterns for enterprise AI agents.
• The same control appears in SOX legacy integration patterns for enterprise AI agents.
• If this is the blocker in front of you: enterprise AI infrastructure review.