AWS has open-sourced a prompt-configured PII detector that can run on different large language models. The strongest reported Core F1 score is 83.1%, which makes the project interesting for experimentation and incomplete as a single compliance control.
The AWS technical article and MIT-0 sample repository were published on September 10, 2026. The package targets messy free text such as support conversations, HR documents and training corpora rather than tidy database fields.
What the AWS PII detector actually contains
| Layer | Implementation | Operational consequence |
|---|---|---|
| Detection | Instruction-driven LLM prompt | Entity scope can change without retraining. |
| Backend | Bedrock Converse adapter or custom inferencer | Teams can compare hosted and self-managed models. |
| Location | Regex offset recovery | Returned text values are mapped back to source positions. |
| Label cleanup | Alias and morphology mapping | Near-miss labels are mapped to the configured vocabulary. |
| Unknowns | UNK label | Unmapped hallucinated labels remain visible instead of being forced into a category. |
The default prompt asks for 15 categories, including private and public names, addresses, contact information, financial data, identifiers, credentials, dates and URLs. The Python package requires Python 3.11 or later, and its only declared runtime dependency is boto3. A custom backend only needs to accept Bedrock-Converse-shaped messages and return a JSON list of detections.
The benchmark covers 49,365 records
AWS evaluated nine LLM-based detectors using five public corpora. The combined sample contains 49,365 records and 222,114 ground-truth core spans across German, English, Spanish, French, Hindi, Italian, Dutch and Telugu.
Labels from every detector and dataset were mapped to 12 shared core entities. A detection counted as correct only when its start position, end position and canonical label exactly matched the ground truth. That strict span rule matters because finding the right digits with the wrong boundary still fails the test.
AWS-reported Core F1 ranges from 74.9% to 83.1%
| Detector backend | Core F1 | Indicative seconds per detection |
|---|---|---|
| Mistral Large 3 on Bedrock | 83.1% | 1.16 |
| OSS-GPT 20B on EC2 | 81.6% | 1.17 |
| PrivacyFilter on EC2 | 80.7% | 2.15 |
| Nova Lite 2 on Bedrock | 74.9% | 0.77 |
These are AWS-reported results, not measurements independently reproduced by MustHave.ai. AWS also says the latency column extrapolates wall-clock batch time back to one record. It is useful for relative orientation, not a guaranteed single-request latency.
An 83.1% F1 score is not an 83.1% privacy guarantee
F1 combines precision and recall. It does not directly tell a compliance team how many sensitive spans escaped detection. The aggregate also combines entity types with different consequences. Missing a public name is not equivalent to missing a password, national identifier or bank account.
AWS reports that OSS-GPT 20B scored above 95% on SSN, financial and ID-number categories in one dataset breakdown, while DATE remained a shared weak spot at about 50%. That variation is the reason production acceptance should be defined per entity and per language, not by one global F1 number.
Prompt customization changes scope without changing weights
Teams can add company names, occupations, crypto-wallet addresses or other domain categories through prompt definitions and examples. This is the core advantage over a fixed-label token classifier. It is also a governance burden: a prompt edit changes what the safety layer claims to detect.
Treat the prompt, label aliases and examples as versioned policy code. Every change should trigger the same language and entity regression suite used for a model change.
A go-live gate for real data
- Measure recall separately for credentials, financial data, national identifiers and contact details.
- Build language-specific tests from authorized examples that resemble production text.
- Review a sample of negative records to estimate false negatives.
- Add deterministic recognizers for stable formats such as payment cards, email addresses and national IDs.
- Send uncertain or high-impact records to a human review queue.
- Store model ID, prompt version, detector version and decision log for every batch.
Where this control belongs
Use the detector as one layer before training, retrieval or analytics, not as proof that a dataset is clean. Our ChatGPT financial-data access analysis explains why access controls and data contracts remain important even when models improve. The UK medical AI lifecycle guide shows the same principle in another high-impact setting: controls have to keep measuring performance after deployment.
The practical verdict
The AWS PII detector is a useful open test bed for configurable, multilingual detection. Its published numbers also make the limitation visible. A production privacy pipeline still needs entity-specific recall gates, deterministic checks, review and auditability.
Primary sources
Checked September 14, 2026. Benchmark scores, latency estimates and category-level results are AWS-reported. Organizations should validate the detector on authorized data from their own languages and workflows.
