{"id":24857,"date":"2019-04-03T14:30:32","date_gmt":"2019-04-03T18:30:32","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=24857"},"modified":"2019-04-04T15:24:52","modified_gmt":"2019-04-04T19:24:52","slug":"utf-8s-history-and-virtues","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2019\/04\/03\/utf-8s-history-and-virtues\/","title":{"rendered":"UTF-8’s History and Virtues"},"content":{"rendered":"

Rob Pike<\/a>:<\/p>\n

What happened was this. We had used the original UTF from ISO 10646\nto make Plan 9 support 16-bit characters, but we hated it. We were\nclose to shipping the system when, late one afternoon, I received a\ncall from some folks, I think at IBM - I remember them being in Austin\n- who were in an X\/Open committee meeting. They wanted Ken and me to\nvet their FSS\/UTF design. We understood why they were introducing a\nnew design, and Ken and I suddenly realized there was an opportunity\nto use our experience to design a really good standard and get the\nX\/Open guys to push it out. We suggested this and the deal was, if we\ncould do it fast, OK. So we went to dinner, Ken figured out the\nbit-packing, and when we came back to the lab after dinner we called\nthe X\/Open guys and explained our scheme. We mailed them an outline\nof our spec, and they replied saying that it was better than theirs (I\ndon’t believe I ever actually saw their proposal; I know I don’t\nremember it) and how fast could we implement it? I think this was a\nWednesday night and we promised a complete running system by Monday,\nwhich I think was when their big vote was.<\/p>

So that night Ken wrote packing and unpacking code and I started\ntearing into the C and graphics libraries. The next day all the code\nwas done and we started converting the text files on the system\nitself. By Friday some time Plan 9 was running, and only running,\nwhat would be called UTF-8. We called X\/Open and the rest, as they\nsay, is slightly rewritten history.<\/p>

Why didn’t we just use their FSS\/UTF? As I remember, it was because\nin that first phone call I sang out a list of desiderata for any such\nencoding, and FSS\/UTF was lacking at least one - the ability to\nsynchronize a byte stream picked up mid-run, with less that one\ncharacter being consumed before synchronization. Becuase that was\nlacking, we felt free - and were given freedom - to roll our own.<\/p><\/blockquote>\n\n

Ken Thompson<\/a>:<\/p>\n

Below are the guidelines that were used in defining the UCS\ntransformation format:<\/p>\n

1) Compatibility with historical file systems:<\/p>\n

Historical file systems disallow the null byte and the ASCII\n\tslash character as a part of the file name.<\/p>\n

\t2) Compatibility with existing programs:<\/p>\n

\tThe existing model for multibyte processing is that ASCII does\n\tnot occur anywhere in a multibyte encoding. There should be\n\tno ASCII code values for any part of a transformation format\n\trepresentation of a character that was not in the ASCII\n\tcharacter set in the UCS representation of the character.<\/p>\n

\t3) Ease of conversion from\/to UCS.<\/p>\n

\t4) The first byte should indicate the number of bytes to\n\tfollow in a multibyte sequence.<\/p>\n

\t5) The transformation format should not be extravagant in\n\tterms of number of bytes used for encoding.<\/p>\n

\t6) It should be possible to find the start of a character\n\tefficiently starting from an arbitrary location in a byte\n\tstream.<\/p><\/blockquote>\n\n

Rich Felker<\/a>:<\/p>\n

\n

Not only do ASCII bytes never appear in multibyte UTF-8 chars; NO character is ever a substring of another character.<\/p>\n

UTF-8 was really a work of brilliance, guaranteeing what’s pretty much a maximal set of important desirable properties like this.<\/p>\n

Of course the desirable properties necessitate one property that’s hard to like: not all byte sequences can be legal\/valid.<\/p>\n<\/blockquote>\n\n

See also: The History of Unix<\/a> (via Hacker News<\/a>).<\/p>\n\n

Previously:<\/p>\n