Character encoding

In computing, a character encoding is used to represent a repertoire of characters by some kind of an encoding system.^[1] Depending on the abstraction level and context, corresponding code points and the resulting code space may be regarded as bit patterns, octets, natural numbers, electrical pulses, etc. A character encoding is used in computation, data storage, and transmission of textual data. Character set, character map, codeset and code page are related, but not identical, terms.

Early character codes associated with the optical or electrical telegraph could only represent a subset of the characters used in written languages, sometimes restricted to upper case letters, numerals and some punctuation only. The low cost of digital representation of data in modern computer systems allows more elaborate character codes (such as Unicode) which represent more of the characters used in many written languages. Character encoding using internationally accepted standards permits worldwide interchange of text in electronic form.

History

Early binary repertoires include Bacon's cipher, Braille, International maritime signal flags, and the 4-digit encoding of Chinese characters for a Chinese telegraph code (Hans Schjellerup, 1869). Common examples of character encoding systems include Morse code, the Baudot code, the American Standard Code for Information Interchange (ASCII) and Unicode.^[2]

Morse code was introduced in the 1840s and is used to encode each letter of the Latin alphabet, each Arabic numeral, and some other characters via a series of long and short presses of a telegraph key. Representations of characters encoded using Morse code varied in length.

The Baudot code, a five-bit encoding, was created by Émile Baudot in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930.

Fieldata, a six- or seven-bit code, was introduced by the U.S. Army Signal Corps in the late 1950s.

IBM's Binary Coded Decimal (BCD) was a six-bit encoding scheme used by IBM in as early as 1959 in its 1401 and 1620 computers, and in its 7000 Series (for example, 704, 7040, 709 and 7090 computers), as well as in associated peripherals. BCD extended existing simple four-bit numeric encoding to include alphabetic and special characters, mapping it easily to punch-card encoding which was already in widespread use. It was the precursor to EBCDIC.

ASCII was introduced in 1963 and is a seven-bit encoding scheme used to encode letters, numerals, symbols, and device control codes as fixed-length codes using integers.

IBM's Extended Binary Coded Decimal Interchange Code (usually abbreviated as EBCDIC) is an eight-bit encoding scheme developed in 1963.

The limitations of such sets soon became apparent, and a number of ad hoc methods were developed to extend them. The need to support more writing systems for different languages, including the CJK family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ad hoc approaches.

The frustrating dilemma that researchers in this field encountered in the 1980s as they tried to develop universally interchangeable character encodings was that on the one hand, it seemed to be necessary to add more bits to accommodate additional characters. On the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users at the time), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users).

The compromise solution that was eventually hit upon with Unicode, as further explained below, was to break the longstanding assumption (dating back to the old telegraph codes) that each character should always directly correspond to a particular pattern of encoded bits. Instead, characters would be first mapped to an intermediate stage in the form of abstract numbers known as code points. Those code points would then be encoded in a variety of ways and with various default numbers of bits per character (code units) depending upon context. To encode code points higher than the length of the code unit, such as above 256 for 8-bit units, the solution was to implement variable-width encodings where an escape sequence would signal that subsequent bits should be parsed as a higher code point.

Code unit

^[3] Characters / Symbols

Terminology related to code unit:

A character is a minimal unit of text that has semantic value.
A character set is a collection of characters that might be used by multiple languages.

Example: The Latin character set is used by English and most European languages, though the Greek character set is used only by the Greek language.

A coded character set is a character set, where each character is assigned with a unique number.
A code point is a value that can be used in a coded character set. A code point is a 32-bit integer data type, where the lower 21 bits represent a valid code point value and the upper 11 bits are 0.

A code unit is a bit sequence used to encode each single character unit of a repertoire within each encoding form.

Character repertoire (the abstract list of characters):

The character repertoire is an abstract list of more than one million characters found in a wide variety of scripts including Latin, Cyrillic, Chinese, Korean, Japanese, Hebrew, and Aramaic.

Other symbols such as musical notation are also included in the character repertoire. Both the Unicode and GB18030 standards have a character repertoire. As new characters are added to one standard, the other standard also adds those characters, to maintain parity.

The code unit size is equivalent to the bit measurement for the particular encoding:

A code unit in US-ASCII consists of 7 bits;
A code unit in UTF-8 , EBCDIC and GB18030 consists of 8 bits;
A code unit in UTF-16 consists of 16 bits;
A code unit in UTF-32 consists of 32 bits.

Example of a code unit: Imagine a String that contains the letters "abc" followed by the Deseret LONG I, which is represented with two char values. That string contains four characters, four code points, but overall five code units.

To express a character in Unicode, the hexadecimal value is prefixed with the string U+. The valid code point range for the Unicode standard is U+0000 to U+10FFFF, inclusive.

Characters that are in the range U+10000 to U+10FFFF are called supplementary characters. The set of characters from U+0000 to U+FFFF are sometimes referred to as the Basic Multilingual Plane (BMP).

The following table shows code point values examples:

Character	Unicode Code Point	Glyph
Latin A	U+0041	Α
Latin sharp S	U+00DF	ß
Han for East	U+6771	東
Ampersand	U+0026	&
Inverted exclamation mark	U+00A1	¡
Section sign	U+00A7	§

The relationship between code points and code units:

A code point is a character and this is represented by one or more code units depending on the encoding.

In each encoding, the code points are mapped to one or more code units.

The number of code units required to be mapped to a code point varies across encoding forms:

UTF-8

Multiple code units per code point are common in UTF-8 because of the smaller code units. The code points will be mapped to one, two, three, or four code units.

UTF-16

UTF-16 code units are twice as large as 8-bit code units. Therefore, any code points with a scalar value less than U+10000 are encoded with a single code unit.

For code points with a scalar value of U+10000 or higher, two code units are required per code point. These pairs of code units have a unique term in UTF-16: "Unicode surrogate pairs".

UTF-32

The 32-bit code unit used in UTF-32 is large enough that every code point is encoded as a single code unit.

GB18030

Multiple code units per code point are common in GB18030, because of the smaller code units. The code points will be mapped to one, two, or four code units.

Unicode encoding model

Unicode and its parallel standard, the ISO/IEC 10646 Universal Character Set, together constitute a modern, unified character encoding. Rather than mapping characters directly to octets (bytes), they separately define what characters are available, corresponding natural numbers (code points), how those numbers are encoded as a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets. The idea behind this decomposition is to establish a universal set of characters that can be encoded in a variety of ways.^[4] To describe this model correctly one needs more precise terms than "character set" and "character encoding." The terms used in the modern model follow:^[4]

A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, i.e. no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows code pages). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into basic information units. The basic variants of the Latin, Greek and Cyrillic alphabets can be broken down into letters, digits, punctuation, and a few special characters such as the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. Even with these alphabets, however, diacritics pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Ligatures pose similar problems. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations.

A coded character set (CCS) specifies how to represent characters using a number of code points. For example, in a given repertoire, a character representing the capital letter "A" in the Latin alphabet might be assigned to the code point 65, the character for "B" to 66, and so on. Multiple coded character sets may share the same repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different codes. In a coded character set, each code point only represents one character, i.e., a coded character set is a function.

A character encoding form (CEF) specifies the conversion of a coded character set's code points into a set of code units that facilitate storage in a system that represents numbers in binary form using a fixed number of bits (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units would only be able to directly represent code points from 0 to 65,535 in each unit, but larger code points could be represented if more than one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping a single code point from a range of, say, 0 to 1.4 million, to a series of one or more code values from a range of, say, 0 to 65,535.

Next, a character encoding scheme (CES) specifies how the code units should be mapped into an octet sequence suitable for saving on an octet-based file system or transmitting over an octet-based network. There are simple character encoding schemes, like UTF-8, UTF-16BE, UTF-32BE, UTF-16LE or UTF-32LE; compound character encoding schemes, which use byte order marks or escape sequences to switch between several simple schemes (such as UTF-16, UTF-32 and ISO/IEC 2022); and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode).

Although UTF-32BE is a simpler CES, most systems working with Unicode use either UTF-8, which is backward compatible with fixed-width ASCII and maps Unicode code points to variable-width sequences of octets, or UTF-16BE, which is backward compatible with fixed-width UCS-2BE and maps Unicode code points to variable-width sequences of 16-bit words. See comparison of Unicode encodings for a detailed discussion.

Finally, there may be a higher level protocol which supplies additional information that can be used to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML attribute xml:lang.

The Unicode model reserves the term character map for historical systems which directly assign a sequence of characters to a sequence of bytes, covering all of CCS, CEF and CES layers.^[4]

Character sets, maps and code pages

In computer science, the terms "character encoding", "character map", "character set" and "code page" were historically synonymous, as the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units – usually with a single character per code unit. The terms now have related but distinct meanings, reflecting the efforts of standards bodies to use precise terminology when writing about and unifying many different encoding systems.^[4] Regardless, the terms are still used interchangeably, with character set being nearly ubiquitous.

A "code page" usually means a byte-oriented encoding, but with regard to some suite of encodings (covering different scripts), where many characters share the same codes in most or all those code pages. Well known code page suites are "Windows" (based on Windows-1252) and "IBM"/"DOS" (based on code page 437), see Windows code page for details. Most, but not all, encodings referred to as code pages are single-byte encodings (but see octet on byte size.)

IBM's Character Data Representation Architecture (CDRA) designates with coded character set identifiers (CCSIDs) and each of which is variously called a "charset", "character set", "code page", or "CHARMAP".^[4]

The term "code page" does not occur in Unix or Linux where "charmap" is preferred, usually in the larger context of locales.

Contrasted to CCS above, a "character encoding" is a map from abstract characters to code words. A "character set" in HTTP (and MIME) parlance is the same as a character encoding (but not the same as CCS).

"Legacy encoding" is a term sometimes used to characterize old character encodings, but with an ambiguity of sense. Most of its use is in the context of Unicodification, where it refers to encodings that fail to cover all Unicode code points, or, more generally, using a somewhat different character repertoire: several code points representing one Unicode character,^[5] or versa (see e.g. code page 437). Some sources refer to an encoding as legacy only because it preceded Unicode.^[6] All Windows code pages are usually referred to as legacy, both because they antedate Unicode and because they are unable to represent all 2²¹ possible Unicode code points.