Toolkit Labs

Why two identical-looking strings are not equal

Because equality is defined on codepoints and the screen is not. Two strings can render identically and hold different codepoints, and the standard fixes for that - Unicode normalisation and case folding - each solve one slice of it. On 300 constructed strings, 26 are not already in NFC form, 80 change under NFKC, and normalisation fixes 0 of the 30 lookalike cases.

The three problems, kept apart

Normalisation is about the same character having more than one encoding: an accented letter as one codepoint or as a base plus a combining mark. NFC composes, NFD decomposes; both are lossless and reversible in the sense that they round-trip to the same normal form. Compatibility normalisation (NFKC/NFKD) goes further and rewrites characters that are merely similar - a ligature into two letters, a full-width letter into an ASCII one, a superscript digit into a digit. It is lossy on purpose. Confusables are different characters that merely look alike - a Cyrillic letter shaped exactly like an ASCII one. Nothing in normalisation touches those.

What the measurement says

The three traps in those rows

• Validate then normalise gives you a string you never validated. The length changes under NFKC on 47 of these cases, so a limit checked before normalising and a value stored after it are measurements of two different strings. Normalise first, at the boundary, then check.
• Case conversion is not reversible. On 27 cases, upper-then-lower does not return the original, so any "canonical form" built out of .upper() is not canonical. For comparison use casefold(), which exists for exactly this and is not the same function as lower().
• Normalisation is not a security control. It folded 0 of the 30 lookalike cases and removed 0 of the 30 invisible-character cases. A username check that normalises and then compares still lets through a name that is visually identical to somebody else's, and still lets through a name with an invisible codepoint in the middle.

What to do instead

• Pick one form and normalise at the edge. NFC on the way in, everywhere - one place, not per feature. Store the normalised value; keep the raw one only if you must display it exactly.
• Compare with casefold() after normalising, and be explicit that this is caseless matching, not equality.
• Handle invisibles as an explicit rule. Decide which categories of invisible codepoints an identifier may contain, and strip or reject on that rule. Do not expect a normal form to do it: it will not.
• Treat lookalikes as their own problem. That is a confusable-detection question - a skeleton comparison and a script-mixing rule - and it belongs beside your uniqueness check, not inside your normaliser.
• Use NFKC deliberately, or not at all. It is right for a search index and wrong for anything you will show back to a person or use as a key, because it discards distinctions the user typed on purpose.

Where these numbers come from

Every count above is computed with Python's unicodedata on UCD 15.0.0 over UNICODE-300, a corpus of 300 labelled strings built to break naive text handling, in 10 categories of 30 cases - combining marks, joined emoji, astral codepoints, direction controls, zero-width characters, lookalikes, compatibility folds, case mapping and exotic whitespace. Twenty cases and the checker are free and public domain; the full corpus is a paid download. The corpus and the free sample are here.

Where these 300 strings come from

Every number on this page was computed from UNICODE-300: 300 labelled strings, each property measured with unicodedata rather than asserted. The full corpus is EUR 19, one-time; a 20-case slice and the checker are free on the product page, public domain, no account.