Extended ASCII

Output of the program ascii in Cygwin

The term extended ASCII (or high ASCII) refers to eight-bit or larger character encodings that include the standard seven-bit ASCII characters, plus additional characters. The use of the term is sometimes criticized,[1][2][3] because it can be mistakenly interpreted to mean that the ASCII standard has been updated to include more than 128 characters or that the term unambiguously identifies a single encoding, both of which are not the case.

There are many extended ASCII encodings (more than 220 DOS and Windows codepages). EBCDIC ("the other" major 8-bit character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.

Motive

ASCII was designed in the 1960s for teleprinters and telegraphy, and some computing. Early teleprinters were electromechanical, having no microprocessor and just enough electromechanical memory to function. They fully processed one character at a time, returning to an idle state immediately afterward. They were typewriter-derived impact printers, and could only print a fixed set of glyphs, which were cast into a metal type element or elements. Seven-bit ASCII improved over prior five- and six-bit codes: it added the lowercase letters of the English alphabet and more symbols, for a total of 95 printing characters; it encoded the letters in alphabetical order rather than keyboard position; and it used the extra bits to address all characters rather than use control characters to select banks of characters.

ASCII encodes only 95 carefully selected printable characters (94 glyphs and one space), which include the English alphabet (uppercase and lowercase), digits, and 31 punctuation marks and symbols: all of the symbols on a standard US typewriter plus a few; enough to communicate text information. The number of printable characters was deliberately kept small, to keep teleprinters and line printers inexpensive. Some popular peripherals only implemented a 64-printing-character subset: Teletype Model 33 could not transmit "a" though "z" or five less-common symbols (` { | } and ~). and when they received such characters they instead printed "A" through "Z" (forced all caps) and five other mostly-similar symbols (@ [ \ ] and ^).

The ASCII character set is barely large enough for general use, and far too small for universal use. Many more letters and symbols are desirable, useful, or required to directly represent letters of alphabets other than English, more kinds of punctuation and spacing, more mathematical operators and symbols (× ÷ · ≠ ≥ ≈ π etc.), some unique symbols used by some programming languages, ideograms, logograms, box-drawing characters, etc.

The biggest problem for computer users around the world was other alphabets. ASCII's English alphabet almost accommodates European languages, if accented letters are replaced by non-accented letters or two-character approximations. Languages with dissimilar basic alphabets could use transliteration. Users were not comfortable with such compromises.

Modified variants of 7-bit ASCII appeared promptly, trading some lesser-used symbols for highly-desired symbols or letters, such as replacing "#" with "£" on UK Teletypes, "\" with "¥" in Japan or "₩" in Korea, etc. At least 29 variant sets resulted. 12 code points were modified by at least one modified set, leaving only 82 "invariant" codes.

There was always temptation to make a larger character set (preferably by extending a standard character set), but there was no easy path forward. There was no way to define a universal extension. There was no hardware to print or display an extended set. For years, applications were designed around the 64-character set and/or the 95-character set, so several characters acquired new uses. (For example, ASCII lacks "÷", so most programming languages use "/" (and sometimes "\") to indicate division, and most still do not recognize "÷". C interprets the two-character sequences "<%" and "%>" as equivalent to "{" and "}" because "{" and "}" are not available on all keyboards.)

The English alphabet was generally useful on computers around the world, for programming, operation, and international communication, so extended character sets that added to the original ASCII set were a good idea.

Seven-bit ASCII encodes only 128 characters (including control characters). When computers and peripherals standardized on eight-bit bytes, it became obvious that computers and software could handle text that uses 256-character sets at almost no additional cost in programming, and no additional cost for storage. (Assuming that the unused 8th bit of each byte was not reused in some way, such as error checking, Boolean fields, or packing 8 characters into 7 bytes.) An eight-bit character set (using one byte per character) encodes 256 characters, so it can include ASCII plus 128 more characters. (Variable multi-byte schemes based on ASCII can add many more additional characters. Unmodified code will provide valid results searching for sequences of ASCII and extended characters. But software must be more sophisticated, only accepting or creating valid multi-byte characters, and always treating multi-byte characters as one character whenever counting, copying, comparing, inserting, or deleting characters.)

Many manufacturers devised 8-bit character sets consisting of ASCII plus up to 128 of the unused codes, with varying degrees of cooperation or coordination by national and international standards bodies.

128 additional characters is not enough to cover all purposes, all languages, or even all European languages, so the emergence of many proprietary and national ASCII-derived 8-bit character sets was inevitable.

Drawbacks of multiple 8-bit character sets are that conversion of text data from one character encoding to another (transcoding) is complex, and almost always has the possibility of changing non-ASCII characters in the text. Conversion requires a custom map from one character set to the other, or a description of both character sets from which the map can be generated. If any characters in the text are not defined in the destination character set, those characters cannot be converted. Even one unconvertible character can mean failure of conversion, or the converter can substitute a similar character (or characters) or a warning character (typically "?"), and may issue a warning when done. When characters are substituted, a human should inspect and possibly modify the output text, possibly examining the input text.

Proprietary extensions

Various proprietary modifications and extensions of ASCII appeared on non-EBCDIC mainframe computers and minicomputers, especially in universities.

Atari and Commodore home computers added many graphic symbols to their non-standard ASCII (Respectively, ATASCII and PETSCII, based on the original ASCII standard of 1963).

The TRS-80 home computer additions included 64 semigraphics characters (0x80 through 0xAF) that implemented low-resolution block graphics. (Each block-graphic character displayed as a 2x3 grid of pixels, with each block pixel effectively controlled by one of the lower 6 bits.)

IBM introduced eight-bit extended ASCII codes on the original IBM PC and later produced variations for different languages and cultures. IBM called such character sets code pages and assigned numbers to both those they themselves invented as well as many invented and used by other manufacturers. Accordingly, character sets are very often indicated by their IBM code page number. In ASCII-compatible code pages, the lower 128 characters maintained their standard US-ASCII values, and different pages (or sets of characters) could be made available in the upper 128 characters. DOS computers built for the North American market, for example, used code page 437, which included accented characters needed for French, German, and a few other European languages, as well as some graphical line-drawing characters. The larger character set made it possible to create documents in a combination of languages such as English and French (though French computers usually use code page 850), but not, for example, in English and Greek (which required code page 737).

Apple Computer introduced their own eight-bit extended ASCII codes in Mac OS, such as Mac OS Roman.

Digital Equipment Corporation developed the Multinational Character Set, which had fewer characters but more letter and diacritic combinations, based on draft versions of ISO 8859. It was supported by the VT220 and later DEC computer terminals.

ISO 8859 and proprietary adaptations

Eventually, ISO released this standard as ISO 8859 describing its own set of eight-bit ASCII extensions. The most popular is ISO 8859-1, also called ISO Latin 1, which contained characters sufficient for the most common Western European languages. Variations were standardized for other languages as well: ISO 8859-2 for Eastern European languages and ISO 8859-5 for Cyrillic languages, for example.

One notable way in which ISO character sets differ from code pages is that the character positions 128 to 159, corresponding to ASCII control characters with the high-order bit set, are specifically unused and undefined in the ISO standards, though they had often been used for printable characters in proprietary code pages, a breaking of ISO standards that was almost universal.

Microsoft later created code page 1252, a compatible superset of ISO 8859-1 with extra characters in the ISO unused range. Code page 1252 is the standard character encoding of western European language versions of Microsoft Windows, including English versions. ISO 8859-1 is the common 8-bit character encoding used by the X Window System and most Internet standards.

Character set confusion

The meaning of each extended code point can be different in every encoding. In order to correctly interpret and display text data (sequences of characters) that includes extended codes, hardware and software that reads or receives the text must use the specific extended ASCII encoding that applies to it. Applying the wrong encoding causes irrational substitution of many or all extended characters in the text.

Software can use a fixed encoding selection, or it can select from a palette of encodings by defaulting, checking the computer's nation and language settings, reading a declaration in the text, analyzing the text, asking the user, letting the user select or override, and/or defaulting to last selection. When text is transferred between computers that use different operating systems, software, and encodings, applying the wrong encoding can be commonplace.

Because the full English alphabet and the most-used characters in English are included in the seven-bit code points of ASCII, which are are common to all encodings (even most proprietary encodings), English-language text is less damaged by interpreting it with the wrong encoding, but text in other languages can display as mojibake (complete nonsense). Because many Internet standards use ISO 8859-1, and because Microsoft Windows (using the code page 1252 superset of ISO 8859-1) is the dominant operating system for personal computers today, unannounced use of ISO 8859-1 is quite commonplace, and may generally be assumed unless there are indications otherwise.

Many communications protocols, most importantly SMTP and HTTP, require the character encoding of content to be tagged with IANA-assigned character set identifiers.

Multi-byte character encodings

Some multi-byte character encodings (character encodings that can handle more than 256 different characters) are also true extended ASCII. That means all ASCII characters are encoded with a single byte with the value that is used in ASCII to encode that character. They can be used in file formats where only ASCII bytes are used for keywords and file format syntax, while bytes 0x80-0xFF might be used for free text, including most programming languages, where language keywords, variable names, and function names must be in ASCII, but string constants and comments can use non-ASCII characters. This makes it much easier to introduce a multi-byte character set into existing systems that use extended ASCII.

UTF-8 is true extended ASCII, as are some Extended Unix Code encodings.

ISO/IEC 6937 is not extended ASCII because its code point 0x24 corresponds to the general currency sign (¤) rather than to the dollar sign ($), but it is an extended version of the International Reference Version of ISO 646.

Shift JIS is not true extended ASCII, because although ASCII characters are encoded the same as in ASCII (except for the backslash; its position is used for the yen character), multi-byte characters can also include ASCII bytes (0x00-0x7F). Shift JIS can directly be used in programming languages and languages such as HTML (if accepting that backslash looks like ¥), because the bytes used for free text delimiters are not used as part of non-ASCII characters. UTF-16 is even less extended ASCII because ASCII characters are stored as two bytes with one byte equal to 0x00. Porting an existing system to support character sets as Shift JIS or UTF-16 is complicated and bug prone.

Usage in computer-readable languages

For programming languages and document languages such as C and HTML, the principle of Extended ASCII is important, since it enables many different encodings and therefore many human languages to be supported with little extra programming effort in the software that interprets the computer-readable language files.

The principle of Extended ASCII means that:

See also

References

  1. ↑ Benjamin Riefenstahl (26 Feb 2001). "Re: Cygwin Termcap information involving extended ascii charicters". cygwin (Mailing list).
  2. ↑ S. Wolicki (Mar 23, 2012). "Thread: Print Extended ASCII Codes in sql*plus".
  3. ↑ Mark J. Reed (March 28, 2004). "vim: how to type extended-ascii?". Newsgroup: comp.editors.

External links

This article is issued from Wikipedia - version of the Thursday, May 05, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.