How well Painfold finds personal data
Out of 100 names in your feedback, Painfold removes about 97. That's why you review every finding before sharing.
- Names: 97% are removed in German and English feedback, and at least 92% in every one of 13 name-origin groups.
- Emails, phone numbers, IBANs and the other kinds of personal data: 96% of the values in hand-written hard cases are removed.
- Stage 2 target, at least 90% of names removed in every group: met.
How it's measured
Painfold is measured on made-up feedback with known personal data, in German and English: hand-written hard cases, hand-written names typical of 13 communities living in Austria, long texts, and generated messages.
The name model is measured in a browser, the way it runs for you. Its rules were tuned on a separate set of sentences; the test sentences were checked only once, after tuning. They were written by the same person who tuned the rules, so a test set written by someone else is planned.
Removed: the value is gone from the cleaned file, whatever kind of data it was taken for. Removed as a name: removed and recognised as a name.
Names
325 test names
- Removed
- 97%
- Removed as a name
- 97%
- Removed without the first-name list
- 90%
271 of 325 test names contain a first name from Painfold's list. The list is used only right after a greeting, a thank-you, a sign-off or an introduction ("hi anna", "LG Markus").
By name-origin group
Target 90%
The groups exist only in the test data, so that names from every community are measured. Painfold never infers anyone's origin.
- German-speaking 100%As a name 100%, without the list 92%
- English 92%As a name 92%, without the list 88%
- Bosnian, Serbian, Croatian 96%As a name 96%, without the list 92%
- Turkish 100%As a name 100%, without the list 92%
- Romanian 100%As a name 100%, without the list 92%
- Hungarian 96%As a name 96%, without the list 92%
- Polish 92%As a name 92%, without the list 92%
- Ukrainian 100%As a name 100%, without the list 88%
- Arabic 100%As a name 100%, without the list 92%
- Afghan 100%As a name 100%, without the list 92%
- African 92%As a name 88%, without the list 84%
- South Asian 96%As a name 96%, without the list 88%
- East Asian 100%As a name 100%, without the list 92%
Names in long texts
The name model reads about 300 words at a time, so longer texts are checked in overlapping parts. These texts are four to five times that long.
- At the start
- 100%
- In the middle
- 100%
- At the end
- 100%
Other personal data
Hand-written hard cases: values written in unusual ways, pasted or disguised, and look-alikes that must not be removed, such as order numbers.
| Kind of data | Found | Correct | Values |
|---|---|---|---|
| Email addresses | 90% | 100% | 10 |
| Phone numbers | 100% | 100% | 11 |
| IBANs | 100% | 100% | 7 |
| Card numbers | 100% | 100% | 6 |
| Social security numbers | 100% | 86% | 6 |
| VAT IDs | 100% | 100% | 7 |
| Links | 100% | 100% | 5 |
| IP addresses | 100% | 100% | 3 |
| Dates of birth | 100% | 100% | 6 |
| Street addresses | 86% | 100% | 7 |
| Postcodes | 100% | 100% | 6 |
| Names | 67% | 100% | 3 |
| Places | 100% | 100% | 7 |
Found: the share of values removed as the right kind of data. Correct: the share of findings that really were that kind of data.
Generated data (a check that nothing broke)
Messages built from the same templates as the sample file. The detectors could be tuned to these sentences, so the numbers are high by design: they show that nothing broke.
| Kind of data | Found | Correct | Values |
|---|---|---|---|
| Email addresses | 100% | 100% | 170 |
| Phone numbers | 100% | 100% | 157 |
| IBANs | 100% | 100% | 100 |
| Card numbers | 100% | 100% | 100 |
| Social security numbers | 100% | 100% | 141 |
| VAT IDs | 100% | 100% | 100 |
| Links | 100% | 100% | 100 |
| IP addresses | 100% | 100% | 125 |
| Dates of birth | 100% | 100% | 167 |
| Street addresses | 100% | 100% | 176 |
| Postcodes | 100% | 100% | 176 |
| Names | 95% | 100% | 361 |
| Places | 87% | 98% | 176 |
Known gaps
- Nicknames after a greeting or thank-you that are not on the first-name list, such as "cheers ollie".
- Names at the start of a sentence, with nothing before them: "Sanne van der Berg says…".
- Some particles and multi-part names: "Saif ur Rehman", "Tesfaye Bekele".
- Small towns: in "Mehmet Aydın aus Wels" the name is removed, but not "Wels".
- Email addresses spelled out to fool spam filters ("name AT example DOT net").
- English addresses that start with a flat number ("Flat 2, 14 Market Street").
- A few document numbers that look like an Austrian social security number are removed although they are not one.
Anything missed can be marked in the review with M.
Speed
A file with 5,000 rows is checked for emails, phone numbers and the other rule-based data in about 1.5 seconds. With the name model in your browser, names in it take 70 to 115 seconds, checked in the background while you review.
The name model
Names and places are found by Davlan/distilbert-base-multilingual-cased-ner-hrl (AFL-3.0), a multilingual model trained on news text, including the CoNLL-2003 data set. The CoNLL texts are themselves licensed, which is a grey area for models trained on them. Painfold serves the model from its own site and runs it in your browser.