Sending Sensitive Data to an AI Service: De-identification That Fails Closed

On This Page

1. The Two Questions Teams Collapse Into One
2. Why a Routing Setting Is Usually Not a Control
3. The Fail-Open Pattern, in Two Places
4. The Pattern That Works: Scrub, Then Gate
5. What a De-identifier Has to Remove
6. Coverage Is Not Effectiveness
7. Deliberate Exceptions, Done Properly
8. The Part Everyone Forgets: What You Keep
9. Making It Auditable
10. Design Checklist
11. Frequently Asked Questions
12. Conclusion

1The Two Questions Teams Collapse Into One

When an application sends user content to a language model — support transcripts, uploaded documents, call recordings, case notes — there are two independent questions, and teams routinely answer one and believe they have answered both.

The contractual answer is the one that gets attention, because it is the one with a document attached. The technical answer is the one that decides what actually happens at three in the morning when a scheduled job runs. A signed agreement covering a specific service does nothing for a code path that calls a different service, and a good de-identifier does nothing on a path that doesn't call it.

2Why a Routing Setting Is Usually Not a Control

The common design is a configuration flag that selects between a covered endpoint and a general one — some variant of USE_COVERED_ENDPOINT. It is a reasonable idea and it becomes a control only if every call path consults it. In practice, applications accumulate model calls in a lot of places: scheduled jobs, request handlers, one-off scripts, analysis modules, a helper someone wrote for a report.

The measurement is straightforward and worth doing before you trust the flag:

# How many places construct a model client?
git grep -n "anthropic.Anthropic(\|OpenAI(\|new OpenAI(" | wc -l

# How many read the flag that is supposed to govern them?
git grep -n "USE_COVERED_ENDPOINT" | wc -l

# If the second number is 1 and the first is 30, the setting
# governs one code path and documents intent for the other 29.

A large gap between those two numbers is extremely common, and it is not a sign of a careless team. It is what happens when a policy is introduced after the integrations already exist: the flag gets added where someone remembered, and every subsequent call site is written by copying an existing one that predates the rule.

The structural fix is to remove the opportunity to forget. Route every model call through one function that selects the destination, and make the raw client constructor something that is not available to call directly — enforced by a lint rule or an import restriction, not by a convention in a README.

3The Fail-Open Pattern, in Two Places

Where controls of this kind break, they almost always break open. Two instances appear so reliably that they are worth checking for by name.

Fail-open 1: the fallback on the covered client

if use_covered and credentials_present:
    try:
        client = CoveredClient(...)          # the permitted path
    except Exception:
        pass                                 # ...and on to the next line

client = DirectClient(api_key=api_key)       # NOT the permitted path

A caller that explicitly requested the covered service is silently given the other one whenever the covered one is unavailable — a credential problem, an account still pending verification, a transient outage. The failure is invisible because the job still produces its output. A control that degrades to the non-compliant path under load or misconfiguration is worse than no control, because it produces exactly the conditions under which nobody is watching.

The correct behaviour is to stop. A report that doesn't run is recoverable; a disclosure isn't.

Fail-open 2: the check that returns "clean" when it cannot run

The second instance is subtler and appears in de-identification pipelines specifically. A verification step returns a list of anything it still found in the scrubbed text — empty means clean. If that step depends on an optional component, such as a language model for entity recognition, the natural implementation returns an empty list when the component is missing:

def leakage_check(text):
    """Returns [] if the gate passed (safe to send)."""
    if nlp_model is None:
        return []          # <-- indistinguishable from "found nothing"
    ...

# The caller cannot tell these apart:
#   []  = "I checked and it is clean"
#   []  = "I could not check at all"

4The Pattern That Works: Scrub, Then Gate

A de-identification step that is trusted without verification is a single point of failure. The pattern worth building is two stages, where the second checks the output of the first independently:

1. Scrub. Remove identifiers by pattern and by entity recognition. Return both the cleaned text and an audit record of what was removed, by category and count.
2. Gate. Run an independent pass over the already-cleaned text looking for anything that still looks like an identifier. Treat a non-empty result — or an inability to check — as a refusal to send.

The gate matters because the scrub stage is pattern-based and pattern-based redaction has a predictable blind spot: it removes what it was told to look for. A name in an unusual position, an identifier in an unexpected format, a phone number written in words — these survive the scrub and are caught, if at all, by a second pass that asks a different question. The two stages fail differently, which is the entire point of having both.

Where multiple paths send content to a model, the scrub-and-gate pair should be applied at all of them, not just the one with a user interface attached. It is common to find a document-upload path implementing the complete pattern correctly while the scheduled job that processes far more sensitive content scrubs without gating — because the upload path had a screen where a warning could be displayed, and the batch job did not.

5What a De-identifier Has to Remove

If you operate under a specific regime, use its list — the HIPAA Safe Harbor method, for example, enumerates eighteen identifier categories and is a usable specification rather than a principle. Independent of regime, these are the categories that get missed:

• Names in free text. The hardest case and the most important. Pattern matching alone cannot find them; this is what entity recognition is for, and it is why the optional-component problem above matters so much.
• Names by relationship phrasing. "her sister Anna," "the patient's father" — catchable by pattern, and frequently not attempted.
• Dates. Not just dates of birth. Admission dates, appointment dates, and any date more specific than a year are identifiers under Safe Harbor, and they are the category teams most often argue about and most often get wrong.
• Ages over 89. A specific carve-out that a generic redactor will not know about.
• Geographic detail below state level, including street addresses. Postal codes are usually handled; street addresses frequently are not, because they have no reliable pattern and entity recognition is often configured to return only person entities.
• Record, account, policy and member numbers. Often catchable as long digit runs, but the keyword-led forms ("Group Number: 4482-A") need their own rule.
• Anything in the model's output. If the model echoes content back and you store or display it, the output needs the same treatment as the input.

A related trap: a domain-specific vocabulary will be mistaken for names by a general entity recognizer. Drug names, treatment programme abbreviations and clinical terms get tagged as people, which produces both noise and, worse, a false sense that the recognizer is working hard. A whitelist of domain vocabulary is a real part of the design, not a nicety.

6Coverage Is Not Effectiveness

These are two different claims, and only one of them can be established by reading code:

• Coverage: which code paths call the de-identifier, and which of those also gate the result. Answerable by static review, completely and cheaply.
• Effectiveness: how much the de-identifier actually removes when run against realistic input. Answerable only by running it and counting what survives.

Teams and assessors alike tend to stop after coverage, because coverage produces a satisfying table. But a de-identifier nobody has measured is a control nobody can rely on. The measurement is not exotic: assemble a corpus of realistic inputs with known identifiers planted in them, run the pipeline, and count what comes through. Record the result and the date.

7Deliberate Exceptions, Done Properly

Not every path should use the full de-identifier, and treating every exception as a defect is its own kind of error. A summary intended for internal staff may need the names and dates that Safe Harbor removes — stripping them would destroy the thing the feature exists to produce.

What distinguishes a considered exception from an oversight is entirely visible in the code:

• A comment at the call site stating that the full de-identifier is deliberately not used, and why.
• A narrower, purpose-built removal step applied instead — not nothing.
• That step applied in both directions: to the input before the model sees it, and to the model's output before it is stored or displayed.
• A correspondingly narrower destination — a covered endpoint, a shorter retention period, tighter access.

When you find one of these while reviewing a system, record it as a control rather than a gap. It is evidence that the team reasoned about the problem, and it is worth distinguishing from the paths that simply never called anything.

8The Part Everyone Forgets: What You Keep

De-identification governs what leaves your system. It says nothing about what accumulates inside it. An AI feature that processes sensitive content almost always produces stored artifacts, and those are frequently the largest concentration of sensitive data in the application:

• A cache of source content, kept so a job doesn't re-fetch or re-process it.
• Model inputs and outputs written to a log or a file for debugging, then never removed.
• A queue or staging file holding items awaiting processing.
• Intermediate transcripts or extracted text, often keyed by an identifier that makes them trivially re-linkable.

Three questions settle this: what accumulates, who can write to it, and what removes it. The third is the one that is usually answered by "nothing," because deletion is never the feature anyone was asked to build. A retention period that exists only as an intention is not a retention period — it needs to be a scheduled task, and the scheduler already exists in any application sophisticated enough to have this problem.

Worth checking specifically: the access control on any endpoint that writes into such a store. An ingestion endpoint guarded by a shared internal key is a route for fabricated content to enter a pipeline that will summarize it and attach the result to a real record. That is an integrity problem rather than a disclosure one, and it is easy to overlook precisely because the data is flowing inward.

9Making It Auditable

The most useful artifact this kind of system can produce is a record of which destination handled each request. Write, for every model call: the timestamp, the destination service, the code path that initiated it, and whether de-identification and gating were applied. Not the content.

That log answers, definitively and from your own records, the question that otherwise gets answered by inference: has sensitive content ever gone somewhere it shouldn't have? It converts a question about intent into a question about fact, and it is far more convincing than any assurance about how the code is supposed to behave. It is also the first thing worth building, because it tells you whether you have an active problem or a theoretical one.

Retain and protect that log deliberately. It is the one accumulating artifact in this design that should not be aggressively trimmed, because it is the audit trail — and in a regulated context an audit trail is a control in its own right.

10Design Checklist

Routing

• Every model call goes through one function; constructing a client directly is prevented, not discouraged.
• The count of call sites and the count of sites reading the routing setting have been compared, and match.
• When the permitted destination is unavailable, the call fails rather than falling through.

De-identification

• Every path sending free text to a model scrubs it first.
• Every path also gates the scrubbed result before sending.
• The gate distinguishes clean, not clean, and could-not-determine — and the third stops the send.
• Model output is treated with the same care as model input where it is stored or displayed.
• Effectiveness has been measured against realistic input, with the result and date recorded.

Storage and Audit

• Every accumulating artifact is enumerated, with a retention period implemented as a scheduled task.
• Endpoints that write into content stores have their own credentials, not a shared internal key.
• A per-call record of destination and treatment exists, is retained, and is protected.

11Frequently Asked Questions

12Conclusion

Sending sensitive content to a language model raises two separate questions — whether the destination is permitted to receive it, and whether it is de-identified before it goes — and a system can pass one while failing the other completely. The failures are consistent: a routing setting that most call paths never consult, a fallback that quietly selects the non-permitted destination, and a verification step that reports success when it could not run.

The design that holds up is unglamorous. One function that every model call goes through. A scrub stage and an independent gate, where inability to check is a refusal rather than a pass. Deliberate exceptions documented at the call site with a compensating control. A retention period on everything that accumulates. And a per-call record of where each request went, so the question can be answered from evidence rather than from intent.

For the adjacent problem of code produced by AI assistants rather than data sent to them, see How to Secure AI-Generated Code Before You Ship It. For the credential handling these integrations depend on, see Hardcoded Secrets and Verifying a Credential Rotation Actually Took Effect.