UNICODE-300 — 300 strings that break naive text handling

UNICODE-300 is a labelled corpus of 300 short strings, 30 in each of ten categories, built from a named inventory of Unicode building blocks: combining marks, ZWJ emoji sequences, codepoints beyond the BMP, variation selectors, bidi controls, zero-width characters, Cyrillic and Greek homoglyphs, NFKC compatibility characters, case-mapping oddities, and whitespace that is not U+0020.

It is for the code that stores, truncates, slugifies, compares or validates text a stranger typed — the place where len(), a database column limit and a browser disagree about how long a string is.

Every property is measured, not asserted

Each case carries codepoint count, UTF-8 byte length, UTF-16 code-unit length, NFC stability, NFKC length, upper and lower lengths, case round-trip stability, and flags for combining marks, zero-width characters, bidi controls, astral codepoints and variation selectors. All of it is computed by Python's unicodedata at build time, on UCD version 15.0.0, which is recorded in kit.json. The category is the construction rule, so a label cannot be wrong; the numbers are re-derivable by you.

category	cases	what it breaks
`combining`	30	len() counts codepoints, not what a reader sees: truncating here can leave a combining mark attached to the wrong letter, or standing alone.
`emoji_zwj`	30	one visible glyph, several codepoints joined by U+200D: a per-character loop, a reverse, or a hard character limit splits it into pieces that render as strangers.
`astral`	30	beyond the BMP: one codepoint is two UTF-16 code units, so JavaScript's .length, a UTF-16 column limit and Python's len() disagree about the same string.
`variation_selector`	30	an invisible U+FE0F/U+FE0E decides text or emoji presentation; strip it and the glyph changes, keep it and the length is one more than it looks.
`bidi_control`	30	invisible direction overrides reorder what is displayed without changing what is stored, so a filename or a name can read differently than it compares.
`zero_width`	30	invisible codepoints survive trim(), slip through many alphanumeric checks, and make two identical-looking identifiers unequal.
`homoglyph`	30	Cyrillic and Greek letters that look like ASCII. NFKC does NOT fold them, so normalising is not a defence against a lookalike.
`nfkc_fold`	30	compatibility characters whose NFKC form has a different length, so validating before normalising and storing after gives two different strings.
`case_mapping`	30	case conversion is not length-preserving and not always reversible: upper and lower round-trips change these strings.
`whitespace`	30	spaces and line terminators that are not U+0020 or U+000A: splitlines() breaks on some of them, strip() does not remove others.

The free slice, and the checker

Twenty cases — two per category, the two lowest ids — are public, together with the whole checker. No account, no email address, nothing to agree to.

curl -O https://toolkitlabs.org/unicode300/unicode300-free.zip
unzip unicode300-free.zip && cd unicode300-free
python3 check.py --fn controls:ascii_fold --cases open20.jsonl

check.py runs your own function over every case and counts three things as failures: it raised, it returned a non-string, or it returned a string that cannot be encoded as UTF-8. Everything else is reported as an observation per category, because whether stripping a zero-width character is right depends on what your function is for.

What the three control functions do to all 300

Produced by running check.py over the full corpus at build time — the same command, shipped in both kits, so you can reproduce every number here:

function	changed	zero_width_kept	bidi_kept	astral_lost
`identity`	0	60	30	0
`ascii_fold`	294	0	0	60
`truncate8`	198	60	30	12

identity returns the text unchanged, ascii_fold is the classic NFKD-then-drop-non-ASCII slug step, and truncate8 is a column that holds eight characters. Read the table as what each one did, not as a verdict.

What is in the free download

file	bytes
`README.md`	3,170
`open20.jsonl`	14,387
`check.py`	3,840
`controls.py`	578
`kit.json`	390
`LICENSE-CC0.txt`	467

Zipped: 6,907 bytes. Everything in it is CC0 1.0 — public domain, no attribution required, commercial use included, yours whether you buy anything or not.

The paid tier is the other 280 cases

EUR 19 buys the complete 300-case corpus as JSONL, the same corpus as a flat CSV with one column per measured property, and the same checker and control functions. Nothing is encrypted and nothing is time-limited.

Full corpus — EUR 19

Instant download after payment; digital goods, delivered immediately. Payments are processed by Stripe. Prices include VAT where applicable.

Verifying nothing moved

Corpus SHA-256 e3e1c76e836bd259c571967f0210bc7cd1e081b808720b137ae0cafed934b780, over the 300-line JSONL exactly as shipped. kit.json in both downloads carries the same digest and the UCD version, so you can prove the cases were not changed after these numbers were published.

What this is not

Not a Unicode conformance suite. The UCD's own test files and ICU own that job, they are free, and this corpus does not replace them.
The properties are Python's unicodedata on UCD 15.0.0. Another runtime, or a later UCD, can legitimately report a different value for the same string.
The strings are constructed from an inventory, not sampled from real traffic. The category counts are equal by construction; your users' text is not.
The free slice is two cases per category chosen as the lowest ids — a stratified sample of the same corpus, not independent data.
Delivery is an unlisted download URL shown by Stripe after payment. It is shareable and there is no per-buyer access control. That is the honest description, not a feature.