Why cutting a string to N characters breaks emoji and accents
Because a cut at position n is a cut in the middle of something. Across 300 constructed strings there are 2,680 places you could cut, and 138 of them (5.1%) damage the text: 60 split a combining mark off its letter, 44 leave a joiner dangling with nothing to join, and 34 sever a variation selector from the character it modifies. 94 of the 300 strings have at least one cut position that damages them.
What "damaged" means here
A slice is damaging when the result is text no reader intended and no library will repair:
- An orphaned combining mark. Cut between a letter and its accent and the accent attaches to whatever ends up before it, or floats alone. It is still valid Unicode, which is why nothing raises.
- A dangling zero-width joiner. One visible emoji is often several codepoints joined by U+200D. Cut inside it and one glyph becomes two or three unrelated ones - plus a trailing joiner that some fonts render as a box.
- A severed variation selector. An invisible U+FE0F decides whether the previous character is drawn as an emoji or as monochrome text. Remove it and the glyph changes shape.
- A split surrogate pair. In UTF-16 anything above U+FFFF is two code units. Across the same corpus there are 2,786 code-unit cut positions and 106 of them leave a lone surrogate - a value that cannot be encoded as UTF-8 at all. Downstream that is an exception, a replacement character, or a rejected JSON document.
The naive column
Take the classic "first eight characters" for a preview or a fixed-width column. Of these 300 strings, 198 are longer than eight codepoints; cutting at that point leaves an orphaned combining mark on 6 of them, and the same cut applied to the UTF-16 encoding splits a surrogate pair on 12. A single fixed number, on one corpus, and it is already wrong dozens of times.
What to do instead
- Cut on grapheme boundaries when a person will read the result.
Intl.Segmenterwithgranularity: "grapheme"in JavaScript; in Python the standard library has no segmentation, so this needs a third-party module. Everything else is an approximation you will be debugging later. - If the limit is bytes, back off to a boundary. Cut the UTF-8, then walk
backwards while the last byte is a continuation byte (
0x80-0xBF) and drop the partial sequence. Never hand out a half-encoded byte string. - If you must cut UTF-16, check for a lone surrogate at the seam and step back one unit if you find one. This is the bug that turns into "invalid UTF-8" three services away from where it happened.
- Budget for the ellipsis. Appending one after cutting to the limit puts you over the limit - a bug that only appears at exactly the boundary length.
- Do not truncate for storage at all if you can avoid it. Reject with a clear message instead: a truncated value is a silent data change that nobody can see downstream.
Related on this site: the four different lengths a string has and why two identical-looking strings are not equal.
Where these numbers come from
The census above was produced by walking every cut position of every string in
UNICODE-300, a corpus of 300 labelled strings built to break naive text
handling - 10 categories, 30 cases each, properties computed with Python's
unicodedata on UCD 15.0.0. Twenty cases and the checker are free and public domain;
the full corpus is a paid download. The corpus and the free sample are here.
Where these 300 strings come from
Every number on this page was computed from UNICODE-300: 300 labelled strings, each property measured with unicodedata rather than asserted. The full corpus is EUR 19, one-time; a 20-case slice and the checker are free on the product page, public domain, no account.