Toolkit Labs

Why cutting a string to N characters breaks emoji and accents

Because a cut at position n is a cut in the middle of something. Across 300 constructed strings there are 2,680 places you could cut, and 138 of them (5.1%) damage the text: 60 split a combining mark off its letter, 44 leave a joiner dangling with nothing to join, and 34 sever a variation selector from the character it modifies. 94 of the 300 strings have at least one cut position that damages them.

What "damaged" means here

A slice is damaging when the result is text no reader intended and no library will repair:

The naive column

Take the classic "first eight characters" for a preview or a fixed-width column. Of these 300 strings, 198 are longer than eight codepoints; cutting at that point leaves an orphaned combining mark on 6 of them, and the same cut applied to the UTF-16 encoding splits a surrogate pair on 12. A single fixed number, on one corpus, and it is already wrong dozens of times.

What to do instead

Related on this site: the four different lengths a string has and why two identical-looking strings are not equal.

Where these numbers come from

The census above was produced by walking every cut position of every string in UNICODE-300, a corpus of 300 labelled strings built to break naive text handling - 10 categories, 30 cases each, properties computed with Python's unicodedata on UCD 15.0.0. Twenty cases and the checker are free and public domain; the full corpus is a paid download. The corpus and the free sample are here.

Where these 300 strings come from

Every number on this page was computed from UNICODE-300: 300 labelled strings, each property measured with unicodedata rather than asserted. The full corpus is EUR 19, one-time; a 20-case slice and the checker are free on the product page, public domain, no account.