Unicode guide: Difference between revisions

Revision as of 01:51, 19 March 2022

This is a WIP page, take nothing here as final.

Introduction

Over the past decade it's been increasingly common to see programming languages add Unicode support: Specifically, support for Unicode strings. This is a good step, but it's not nearly complete and often done in a buggy way. Hopefully in this page I can show what's wrong with this approach and provide some solutions.

Just to make it clear: Unicode is only a part of a complete localization framework. Languages do a bunch of other things wrong, but broken Unicode string handling is the topic I'm covering in this page.

Unicode refresher

If you don't understand what Unicode is, I highly recommend reading the following resources in this order:

The Unicode Standard, Version 14.0 chapters 1, 2, 3, 4, 5 and 23
Unicode Technical Reports
Unicode Frequently Asked Questions

You might also find the following tools helpful:

But as a general overview, Unicode defines the following:

A large multilingual set of abstract characters
A database of properties for each character (this includes case mapping)
How to encode characters for storage
How to normalize text for comparison
How to segment text in to characters, words and sentences
How to break text in to lines
How to order text for sorting

Some of these can be tailored by locale-dependant rules. The Unicode Common Locale Data Repository provides locale-specific information that aids in this tailoring.

ASCII-compatible strings

Traditionally programming languages have stuck with using what I call ASCII-compatible strings.

Here's my definition of this string API:

Characters are represented by an 8-bit byte
Values under 128 are ASCII values
Other values are defined by the current locale setting
Strings are arrays of these characters
Matching is done by comparing bytes, no normalization is needed
Classification, sorting and case conversion depends on the current locale setting
Length is the number of bytes

For example, this model is present in:

C and C++
Python 2
Lua
Ruby
PHP
DOS
Unix-like operating systems
Windows APIs

- encoding unaware, utf-8

- multi-byte strings

- wide strings

Unicode strings

- utf-8

- OS APIs

- string APIs make less sense

- locale tagging

- utf8b

- bytestrings

- poorly defined semantics

ICU strings

ICU/Java?

Non-destructive text processing

- clear, unicode definitions

- rich text

- multiple versions

- metadata

- non-reversible

@@ Line 30: / Line 30: @@
 Some of these can be tailored by locale-dependant rules. The [https://cldr.unicode.org/ Unicode Common Locale Data Repository] provides locale-specific information that aids in this tailoring.
-== ASCII strings ==
+== ASCII-compatible strings ==
-- encoding-neutral but really it's ascii
+Traditionally programming languages have stuck with using what I call ASCII-compatible strings.
-- character set
+Here's my definition of this string API:
-- strings
+* Characters are represented by an 8-bit byte
+* Values under 128 are ASCII values
+* Other values are defined by the current locale setting
+* Strings are arrays of these characters
+* Matching is done by comparing bytes, no normalization is needed
+* Classification, sorting and case conversion depends on the current locale setting
+* Length is the number of bytes
-- ops
+For example, this model is present in:
-- OS APIs provide strings
+* C and C++
+* Python 2
+* Lua
+* Ruby
+* PHP
+* DOS
+* Unix-like operating systems
+* Windows APIs
-- simple, english based
+- encoding unaware, utf-8
-- works with ascii-compatible encodings
+- multi-byte strings
-- you don't have to learn anything complicated
+- wide strings
 == Unicode strings ==