Unicode guide

This is a WIP page, take nothing here as final.

Introduction

- frustrated with online tutorials that are out of date, miss important information or focus on the wrong things

- this is my guide to unicode

- should give you a mental model

- should let you navigate reading the standards and official resources

- should help you understand what tools to use when programming

What is Unicode?

Unicode defines the following:

A large multilingual set of abstract characters (known just as 'characters')
Properties for each character
How to encode characters for storage
How to normalize characters in to a canonical format
How to segment text in to words, sentences, lines, and paragraphs
How to map text between different cases
How to order text for sorting
How to match text for searching
How to incorporate Unicode in to regular expressions

Many of these can be further tailored by locale-dependent rules and custom algorithms.

The layered model

- levels of abstraction

- indexing

- sort

- match

- search

- normalize

- serialize

- case map

- properties

- breaking/segmentation

- reversing

Level 1: Bytes

level 1: bytes. you can compare, search, splitting, sorting. your basic unit is the byte

filesystem/unix/C

Level 2: Code units

level 2: code units. your basic unit is the smallest unit of your unicode encoding: a byte for utf-8, a 16-bit int for UTF-16, a 32-bit int for UTF-32. you can compare, search, splitting, sort. to get to this point you have to handle endianness

windows

Level 3: Unicode scalars

level 3: unicode scalars. your basic unit is a number between 0x0 and 0x1fffff inclusive, with some ranges for surrogates not allowed. to get tho this point you have to decode utf-8, utf-16 or utf-32. you can compare, search, split, etc but it's important to note that these are just numbers. there's no meaning attached to them

python

Level 4: Unicode characters

level 4: unicode characters: your basic unit is a code point that your runtime recognizes and is willing to interpret using a copy of the unicode database. results vary according to the supported unicode version. you can normalize, compare, match, search, and splitting, case map strings. locale specific operations may be provided. to get these the runtime needs to check if the characters are supported.

???

Level 5: Segmented text

level 5: unicode texts: your basic unit is a string of unicode characters of some amount, such as a word, paragraph, grapheme cluster. to get these you need to convert from a string of unicode characters with breaking/segmentation rules

swift/raku