Tuesday, August 8, 2023

Unicode Is Harder Than You Think

Marco Cilloni (via Cédric Luthi, Hacker News):

Reading the excellent article by JeanHeyd Meneide on how broken string encoding in C/C++ is made me realise that Unicode is a topic that is often overlooked by a large number of developers. In my experience, there’s a lot of confusion and wrong expectations on what Unicode is, and what best practices to follow when dealing with strings that may contain characters outside of the ASCII range.

[…]

The fact UTF-32 is a fixed-width encoding is only marginally useful, due to grapheme clusters still being a thing. This means that the equivalence between codepoints and rendered glyphs is still not 1:1, just like in UCS-4[…]

[…]

As I previously mentioned, Unicode codepoints can be modified using combining characters, and the standard supports precomposed forms of some characters which have decomposed forms. The resulting glyphs are visually indistinguishable after being rendered, and there’s no limitation on using both forms alongside each other in the same text bit of text[…]Another headache is the fact Unicode also may define special forms for the same letter or group of letters, which are visibly different but understood by humans to be derived from the same symbol.

Previously:

6 Comments RSS · Twitter · Mastodon


One distinction worth making here is the difference between canonical equivalence and compatibility equivalence.

The usual example is é (U+00E9) versus e followed by U+0301. NFC can bring those two representations into a common form, which is useful when comparing text.

But mathematical alphanumeric symbols introduce another layer. For example, 𝐀 (U+1D400) and ordinary A (U+0041) are different characters. NFC preserves that distinction, while NFKC maps the mathematical bold character back to the ordinary Latin letter.

This matters when styled Unicode text is copied into search fields, usernames, filenames, or databases. An application has to decide whether those differences are meaningful or should be normalized away.

There's also the problem of visually similar characters from different scripts. NFKC doesn't solve that in general. A Latin character and a Cyrillic lookalike may appear almost identical while remaining distinct after normalization.

I've been documenting related examples in a Unicode normalization and canonical equivalence reference, particularly the differences between NFC, NFD, and compatibility normalization.

I think the important point is that normalization isn't just a text cleanup operation. It's a design decision about which distinctions an application intends to preserve.


FWIW, Pangram clocks both the above comment as well as the linked article as 100% AI-generated. There have been a bunch of similar comments here recently where a bot posts some rambling thing on an old article and links to recently published AI-generated article.


@Plume I’ve been deleting tons of them, but some slip through the cracks or seem kind of interesting. What do you think the motive is?


Their repo is public, the AGENTS.md has this odd entry:

"Do not alter the Brazil homepage or US money tool when preparing technical library content unless separately requested."

https://github.com/LetrasDiferentess/letrasdiferentess.github.io/blob/0b0ed62a5a67fe30e32a900fe7e7d7c5e4e61f9d/AGENTS.md

I think "US money tool" refers to the fancy text generator; I'm guessing the plan is to monetize it, which is the most optimistic interpretation of what's happening. Maybe "money tool" is some kind of scam. Or maybe there's some hidden text on the website that's being injected into AI crawlers to influence training data or responses in chats.

At any rate, it's some kind of SEO operation.


@Plume I suppose so, even though most links from user-generated content get nofollow these days.


I wonder how much nofollow actually does.

Leave a Comment