Editing Unicode guide

'''This is a WIP page, take nothing here as final.'''

If you've ever tried to learn Unicode you've most likely looked at online tutorial and learning resources. These tend to focus on specific details about how Unicode works instead of the broader picture. 

This guide is my attempt to help you build a mental model of Unicode that can be used to write functional software and navigate the official Unicode standards and resources.

As a disclaimer: I'm just a random person, some of this might be wrong. But hopefully by the end of reading this you should be able to correct me.

Important note: This page uses Unicode characters in various examples. These may display wrong if your browser or screen reader has trouble rendering Unicode text properly. I've tried my best to write this article to avoid relying on the examples for this reason.

== Standards ==
The Unicode standard defines the following:

*A large numeric codespace
*A large multilingual database of characters
*A database of character properties
*How to encode and decode the codespace
* How to normalize equivalent text
* How to map text between different cases
* How to segment text in to words, sentences, lines, and paragraphs
*How to determine text direction
Some portions of the standard may be overridden (also known as 'tailoring') to aid in localization.

The standard is freely available online in the following pieces:

* [https://www.unicode.org/versions/latest/ Unicode Core Specification] chapters 3 (Conformance) and 4 (Character Properties)
* [https://www.unicode.org/errata/ Unicode Updates and Errata]
* [https://www.unicode.org/charts/ Unicode Character Code Charts] 
* [https://www.unicode.org/ucd/ Unicode Character Database]
* [https://www.unicode.org/reports/index.html#annexes Unicode Standard Annexes]
The Unicode Consortium also defines these in separate standards:
*How to order text for sorting
*How to incorporate Unicode in to regular expressions
*How to handle emoji sequences
*How to handle confusable characters and other security concerns
*A repository of shared localization data
These are also freely available online at:
*[https://www.unicode.org/reports/index.html#standards Unicode Technical Standards]
*[https://cldr.unicode.org/index Unicode Common Locale Data Repository]
Policies for stability in these standards can be found at the [https://www.unicode.org/policies/ Unicode Consortium Policies] page.

== Characters ==
Unicode provides two distinct definitions of the term 'character': Abstract characters and encoded characters. When discussing Unicode the term 'character' means an encoded character.

Abstract characters are the units that make up textual data on a computer. These are usually some portion of a written script that has a unique identity independent of Unicode, such as a letter, symbol, accent, logogram, or spacing but they may be something else entirely. The best way to think of these are atoms used to handle text editing, displaying, organization and storage.

Encoded characters are mappings of an abstract character to the Unicode codespace as a code point. This is almost always what people mean by 'character' in Unicode discussion. There's not a one-to-one mapping between abstract and encoded characters: Abstract characters might be mapped multiple times to aid in compatibility with other character sets, they might not be mapped at all and instead represented using a sequence of other encoded characters, or they might not be representable at all and require addition in future Unicode versions.

In addition to having a code point each character has a set of properties that provide information about the character to aid in writing Unicode algorithms. These include things like name, case, category, script, direction, numeric value, and rendering information.

The last point I want to make is a warning: Characters do not correspond to some human identifiable unit of text such as a glyph, letter, phoneme, syllable, vowel or consonant. They are only useful for building higher level abstractions. General text processing should be done with groups of characters and Unicode-aware algorithms.

Some examples of characters are:

* U+00041 "A": [https://util.unicode.org/UnicodeJsps/character.jsp?a=A&B1=Show LATIN CAPITAL LETTER A]
* U+024CB "Ⓥ": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%E2%93%8B&B1=Show CIRCLED LATIN CAPITAL LETTER V]
* U+1F440 "👀": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%F0%9F%91%80&B1=Show EYES]
* U+00942 " ू": [https://util.unicode.org/UnicodeJsps/character.jsp?a=0942 DEVANAGARI VOWEL SIGN UU]
*U+1F1F3: "🇳": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%F0%9F%87%B3+&B1=Show REGIONAL INDICATOR SYMBOL LETTER N]
* U+02028: [https://util.unicode.org/UnicodeJsps/character.jsp?a=2028 LINE SEPARATOR]
* U+0200B: [https://util.unicode.org/UnicodeJsps/character.jsp?a=200B ZERO WIDTH SPACE]

I've linked to the Unicode Utilities page for each character so you can see the character properties.

== Code points ==
The Unicode standard defines a range of integers from 0x0 to 0x10FFFF as the 'Unicode codespace', and defines a code point as a value within this codespace.

The primary purpose of the code points are to address encoded characters, but it also encodes more. There are seven categories of code points:

* Graphic: Assigned to visible characters
* Format: Assigned to invisible formatting characters
* Control: Assigned to characters used in Unicode and non-Unicode protocols and standards
* Private-use: Assigned for interpretation used outside the Unicode standard
* Surrogate: Reserved for UCS-2 compatibility, must not be encoded
* Noncharacter: Reserved for application internal use, not used for open interchange
* Reserved: Not assigned yet, used in future Unicode versions

Each code point belongs to one of these categories. I bring this system up because there's two major implications that stem from it:

The first is that it's not always possible to interpret a code point as an encoded character: It may be from a future version of Unicode, it may be private use and not known to you, or it may not even be a character at all and instead used for application specific processing.

The second is that exchanging code points must be done mindfully: Surrogate code points can not be exchanged using official Unicode encodings, noncharacters are not intended to be interchanged openly, and private use characters require an external agreement outside the standard.

Unicode also defines a sequence of one or more code points as a 'Coded character sequence', or just 'character sequence' for short. Despite this name it may include any valid code point, including noncharacters or reserved code points. It is strictly a sequence of code points.

The best way to think about code points and sequences of them are as opaque building blocks used in Unicode-aware algorithms. Much like encoded characters don't map to the human concept of character, code points don't map to the machine concept of encoded characters or anything higher level.

== Encodings ==
Storing an arbitrary code point requires a 21-bit number. This a problem for a few reasons:

* Modern computers would store this in a 32-bit number
* Storing a load of 32-bit numbers is space inefficient
Modern development environments break encoded text in to sequences of one or more code units:

* Unix strings use 8-bit code units
* Windows strings use 16-bit code units
* Java and JavaScript strings use 16-bit code units

The Unicode standard defines encoding forms that transform between code points and code units:

* UTF-8 which uses 8-bit code units
* UTF-16 which uses 16-bit code units
* UTF-32 which uses 32-bit code units
These encoding forms encode all valid code points except surrogate code points.

The standard then defines encoding schemes that transform between code units and bytes:

* UTF-8 which is the same as its encoding form
* UTF-16LE and UTF-16BE which use different byte orders
* UTF-32LE and UTF-32BE which use different byte orders
* UTF-16 which is either UTF-16LE or UTF-16BE with a byte order mark for detection
* UTF-32 which is either UTF-32LE or UTF-32BE with a byte order mark for detection
I would like to point out that code unit sequences are often not valid Unicode code unit or even Unicode code code units at all, even if a development environment claims strings are UTF-8 or UTF-16. An obvious example of this is Linux strings where the 8-bit code units are arbitrary bytes without a specified encoding but UTF-8 is the most common encoding.

A less obvious one is Windows and JavaScript strings: Their 16-bit code units should encode UTF-16 but don't enforce its validity. These systems are likely not going to change: Validation has a performance penalty and being strict about it would break compatibility with data that is not Unicode compliant.

Be sure to investigate what guarantees your tools give or don't give regarding encoded data. If the guarantees aren't what you need you can always validate the data yourself.

== Algorithms ==

The Unicode standard defines a set of algorithms that interpret text according to the standard.

In general the algorithms cover the following topics:

*Looking up information about a code point
*Breaking text in to smaller pieces
*Changing the case of text
* Normalizing and comparing text
* Sorting and searching text
* Editing and displaying text
These tend to map cleanly to most text processing done with traditional character sets, with the largest change being that most algorithms operate on character sequences rather than individual characters.

The act of associating code points with some kind of behaviour is known as interpretation. This interpretation varies depending on Unicode version, tailoring, and whether the application supports all of Unicode or just specific code points.

This can be a worrying thought as there's a distinct lack of stability: Running code on one system may have a different output to code on another system depending on their Unicode versions and setup. This is a complicated problem without an easy solution short of never exchanging Unicode text with someone else.

While Unicode does have a stability policy, it is aimed at people writing Unicode algorithms, not people using them. My advice is to treat the output of these algorithms as inherently unstable, and read your tool documentation to see what stability it guarantees.

== Normalization ==
An identical sequence of abstract characters may be represented using multiple different encoded character sequences. This can be due to an abstract character being encoded multiple times, or being encodable using multiple other encoded characters.

An easy example is that the ohm symbol may be represented as any of the following:

* U+03A9 "Ω": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%CE%A9&B1=Show GREEK CAPITAL LETTER OMEGA]
* U+2126 "Ω": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%E2%84%A6&B1=Show OHM SIGN]

For a harder example the abstract character "é" may be represented as:

* U+00E9 "é": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%C3%A9&B1=Show LATIN SMALL LETTER E WITH ACUTE]

But it can also be represented as:

* U+0065 "e": [https://util.unicode.org/UnicodeJsps/character.jsp?a=e&B1=Show LATIN SMALL LETTER E]
* U+0301 " ́": [https://util.unicode.org/UnicodeJsps/character.jsp?a=301&B1=Show COMBINING ACUTE ACCENT]

These are all the same abstract character but can be encoded multiple ways. One is a precomposed character, one is two characters: A base character and a combining character. This makes comparing these for equality very difficult.

To solve this Unicode has a normalization algorithm that can transform a coded character sequence in such a way that it ensure all sequences of the same abstract characters are represented by the same coded character sequence. This works in a series of steps:

The first step is decomposition: Each encoded character is recursively mapped to one or more encoded character sequences that are defined to be equivalent. For the most part this uses a mapping defined in the Unicode database, but special rules are required to decompose Hangul syllables. The simple example above of LATIN SMALL LETTER E WITH ACUTE is expanded to two characters: LATIN SMALL LETTER E and COMBINING ACUTE ACCENT.

The second step is re-ordering: Multiple combining characters can be attached to a base character, and often the order is based on how the character is typed. This step re-orders the combining characters to be in a specific order. Doing this step requires an unbounded buffer which can become a security hazard depending on the application. The standard defines the "Stream-Safe Text Process" which limits this step to processing 30 combining characters but creates output that isn't normalized when dealing with uncharacteristically long inputs.

The third step is composition: This step is optional and does the reverse of decomposition as a form of compression. It looks at the new sequence and recursively matches character sequences in it to decomposition mappings. This step excludes many opportunities to compose: Various scripts have specific exclusions, and single encoded characters will not compose to other single encoded characters. As an example of composition, LATIN SMALL LETTER E and COMBINING ACUTE ACCENT is composed back to LATIN SMALL LETTER E WITH ACUTE. As an example of an exclusion, OHM SIGN will decompose to GREEK CAPITAL LETTER OMEGA but not compose back to OHM SIGN.

When describing these steps I glossed over what it means for encoded characters to be equivalent. Unicode defines two forms of equivalent: Canonical and compatibility equivalence. Both of these equivalences require that the encoded characters represent the same abstract character. Compatibility equivalence goes a step further and defines equivalence between encoded characters that have different appearances or behaviours. This usually includes formatting and other ways to write a character, but does not include other variants of the character such as different cases.

These encoded characters are all compatibly equivalent to the digit two:

* U+00032 "2": [https://util.unicode.org/UnicodeJsps/character.jsp?a=2&B1=Show DIGIT TWO]
* U+000B2 "²": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%C2%B2&B1=Show SUPERSCRIPT TWO]
* U+02082 "₂": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%E2%82%82&B1=Show SUBSCRIPT TWO]
* U+02461 "②": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%E2%91%A1&B1=Show CIRCLED DIGIT TWO]
* U+1D7D0 "𝟐": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%F0%9D%9F%90&B1=Show MATHEMATICAL BOLD DIGIT TWO]

Compatibility equivalence combines with canonical equivalence during during the decomposition step in the normalization algorithm. This creates two types of decomposition:

* Canonical decomposition which uses canonical equivalence
* Compatibility decomposition which uses both canonical and compatibility equivalence

With all that in mind Unicode defines the following normalization forms:

* Normalization Form D (NFD) uses canonical decomposition and skips recomposition
* Normalization Form C (NFC) uses canonical decomposition
* Normalization Form KD (NFKD) uses compatibility decomposition and skips recomposition
* Normalization Form KC (NFKD) uses compatibility decomposition

Normalization is stable between Unicode versions after 4.1 (released in 2005):

* Normalized text from an older version stays normalized in the new version
* Normalized text in a new version stays normalized to the older version if it contains only characters assigned in the older version
As a developer you will normally find normalization in code that checks for equality between abstract character sequences read from elsewhere, such as usernames in databases and filenames on filesystems. This procedure is generally unnecessary for comparing text generated or manipulated within an application unless those operations are not deterministic.

I also want to note that compatibility decomposition is only useful in specific text processing tasks: It does not act as a filter for malicious text that intends to look visually identical to other text that uses different abstract characters. Various security tools exist to filter these 'confusables', but these should not be used indiscriminately as they are inherently lossy algorithms.

One example where compatibility equivalence is useful is useful is screen readers: Text that is formatted may be read as their compatibility equivalent values during normal reads of text, with the actual values read out verbosely later if needed.

For full details on the algorithm check out the standard: [https://unicode.org/reports/tr15/ UAX #15: Unicode Normalization Forms]

== Segmentation ==
Code points and character sequences aren't useful in most text processing. Higher level constructs that map to things humans perceive and reason about are required.

Unicode provides a text segmentation algorithm for breaking character sequences in to groups of sentences, words and user-perceived characters. This can be used to implement many common algorithms such as counting user-perceived characters in text, inserting and removing text, or parsing text in to separate components.

As an example, take the following text: "Hi! 👋🏼". It consists of 6 code points:

* U+00048 "H": [https://util.unicode.org/UnicodeJsps/character.jsp?a=H&B1=Show LATIN CAPITAL LETTER H]
* U+00049 "I": [https://util.unicode.org/UnicodeJsps/character.jsp?a=I&B1=Show LATIN CAPITAL LETTER I]
* U+00021 "!": [https://util.unicode.org/UnicodeJsps/character.jsp?a=!&B1=Show EXCLAMATION MARK]
* U+00020: [https://util.unicode.org/UnicodeJsps/character.jsp?a=%20&B1=Show SPACE]
* U+1F44B "👋": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%F0%9F%91%8B&B1=Show WAVING HAND SIGN]
*U+1F3FC " 🏼": [https://util.unicode.org/UnicodeJsps/character.jsp?a=%F0%9F%8F%BC&B1=Show EMOJI MODIFIER FITZPATRICK TYPE-3]

It breaks in to:

* 5 user-perceived characters: "H", "i", exclamation mark, space, and white waving hand
* 4 words: "Hi", exclamation mark, space, and white waving hand
* 2 sentences: "Hi! " (including a trailing space), and white waving hand
The default breaking algorithms do not do any kind of linguistic or locale analysis. Instead they are simple sets of rules designed to be get a useful results given arbitrary text.

Some use cases considered for these rules include:

* Searching and ordering text
* Selecting text at different granularities
* Moving cursors through text
* Inserting and removing text when editing
* Counting occurrences of text elements

These are desirable goals in most computer programs and tolerant of edge cases: A boundary that is slightly wrong to a human usually doesn't matter in these cases as long as it is consistently wrong. For stronger segmentation guarantees these rules can be tailored for a specific applications or discarded entirely in favour of tools like natural language processing.

One type of segmentation gets a lot more attention than the others: User-perceived characters. These are segmented as 'grapheme clusters' and come in two variants: Legacy and extended. Unless you need to deal with backwards compatibility, extended grapheme clusters are the ones to use. Words and sentences are by default made up of grapheme clusters.      

Grapheme clusters are the closest representation you can get to the idea of a single abstract character. Some newer programming languages even default to these as the default abstraction for their strings. This turns out to work fairly well and reduces the difficulty in writing Unicode compliant programs.      

The main downside to this approach is that string operations are no longer guaranteed to be reproducible between program environments and versions. Your Unicode text may be split one way on one system and another way on another, or change behaviour on system upgrade. One real world example of this would be if you're given a giant character sequence of one base character and thousands of combining characters. One system may treat this as one grapheme cluster, another may split it up during normalization in to many grapheme clusters.      

This lack of stability isn't necessarily a bad thing. After all, the world changes and so must our tools. But it needs to be kept in mind for applications that are expecting stability traditional strings provide. A method to serialize sequences of grapheme clusters would help here, instead of having to recompute them based on code points.      

For full details on the algorithm check out the standard: [https://unicode.org/reports/tr29/ UAX #29: Unicode Text Segmentation]

You can experiment with breaks online using the [https://util.unicode.org/UnicodeJsps/breaks.jsp Unicode Utilities: Breaks] tool.

== Line breaking ==
this section should talk about new line characters, paragraphs, records, vertical tabs, etc

more importantly- how to handle them, are these actually recognized, etc

NLF


NEWLINE GUIDELINES:

new lines (ambiguous):

- CR

- LF

- CRLF

- NEL

- VT

- FF

- LS

- PS

newline function

- macos9 earlier: cr

- unix/osx: lf

- windows: crlf

- ebcdic: nel

treated the same on input? distinguished on output

paragraphs/new lines breaking, record separator

- PS

- LS

word processing NLF is PS

simple text NLF is LS

both PS and LS terminate lines

- stop reading at NLF, LS, FF, PS, don't include it

- write should convert NLF/LS/PS to appropriate output

PAGE SEPARATOR

line breaking algorithm: more complex and intended for displaying text

- new line characters

- provides opportunities to line break rather than strict breaks

- not atomic with graphemes

- new line characters

- provides opportunities to line break rather than strict breaks

- not atomic with graphemes

For full details on the algorithm check out the standard: [https://www.unicode.org/reports/tr14/ UAX #14: Unicode Line Breaking Algorithm]

You can experiment with breaks online using the [https://util.unicode.org/UnicodeJsps/breaks.jsp Unicode Utilities: Breaks] tool.

== Abstractions levels ==
=== Level 1: Bytes ===


level 1: bytes. you can compare, search, splitting, sorting. your basic unit is the byte

filesystem/unix/C

utf-8b

=== Level 2: Code units ===
level 2: code units. your basic unit is the smallest unit of your unicode encoding: a byte for utf-8, a 16-bit int for UTF-16, a 32-bit int for UTF-32. you can compare, search, splitting, sort. to get to this point you have to handle endianness

windows

=== Level 3: Unicode scalars ===
level 3: unicode scalars. your basic unit is a number between 0x0 and 0x1fffff inclusive, with some ranges for surrogates not allowed. to get tho this point you have to decode utf-8, utf-16 or utf-32. you can compare, search, split, etc but it's important to note that these are just numbers. there's no meaning attached to them

python

TODO: Code points can be noncharacters or reserved characters, uh-oh

=== Level 4: Unicode characters ===
level 4: unicode characters: your basic unit is a code point that your runtime recognizes and is willing to interpret using a copy of the unicode database. results vary according to the supported unicode version. you can normalize, compare, match, search, and splitting, case map strings. locale specific operations may be provided. to get these the runtime needs to check if the characters are supported.

???

TODO: noncharacters

=== Level 5: Segmented text ===
level 5: unicode texts: your basic unit is a string of unicode characters of some amount, such as a word, paragraph, grapheme cluster. to get these you need to convert from a string of unicode characters with breaking/segmentation rules

swift/raku

== Non-standard encodings ==
other standards:

- GB 18030

previous standards:

- ucs-2

- ucs-4

- utf-1

in memory storage:

- 32-bit integer

- 'runes'

- wtf-8

- utf-8b

- UTF8-C8

- nfg

- python's weirdness

== General mistakes ==
- languages don't let you store all codepoints

- not tagging data with locale/encoding

- relying on locale

- not using markup

- utf8b

- with and encoding isn't that important

- APIs will give you invalid data

- APIs may not check code units

- APIs might not let you handle surrogates

- code units, etc

- uint32

- utf-32

- not grapheme aware: 🇪🇳🇮🇸 -> 🇪🇳 🇮🇸 , fonts will cheaply display as 🇪 🇳🇮 🇸 , grep

- not the same as ligatures

- fonts cursive

- flags

- default/tailored

For example, two individual letters are often two separate      graphemes. When two letters form a ligature, however, they      combine into a single glyph. They are then part of the same      cluster and are treated as a unit by the shaping engine —      even though the two original, underlying letters remain separate      graphemes.

- round trips, invalid unicode, non unicode, confusables

- length

== Further reading ==
I highly recommend reading the following resources:
* [https://www.unicode.org/versions/latest/ Unicode Core Specification]
*[https://www.unicode.org/glossary/ Unicode Glossary]
* [https://www.unicode.org/reports/index.html Unicode Technical Reports]
* [https://www.unicode.org/faq/ Unicode Frequently Asked Questions]

You might also find the following tools helpful:

* [https://util.unicode.org/UnicodeJsps/ Unicode Utilities]
*[https://unicode.org/ucd/ Unicode Character Database]
While writing this page I researched and documented Unicode support in various programming languages. You can see my notes here: [[Unicode guide/Implementations]].
[[Category:Research]]