Shift JIS

Shift JIS

MIME	Shift_JIS
Standard	JIS X 0208 Appendix 1
Language(s)	Japanese

Shift JIS (Shift Japanese Industrial Standards, also SJIS, MIME name Shift_JIS) is a character encoding for the Japanese language, originally developed by a Japanese company called ASCII Corporation in conjunction with Microsoft and standardized as JIS X 0208 Appendix 1. 1.1% of all web pages use Shift JIS in January 2016.^[1]

Description

Shift JIS is based on character sets defined within JIS standards JIS X 0201:1997 (for the single-byte characters) and JIS X 0208:1997 (for the double byte characters). The lead bytes for the double byte characters are "shifted" around the 64 halfwidth katakana characters in the single-byte range 0xA1 to 0xDF. The single-byte characters 0x00 to 0x7F match the ASCII encoding, except for a yen sign (U+00A5) at 0x5C and an overline (U+203E) at 0x7E in place of the ASCII character set's backslash and tilde respectively. The single-byte characters from 0xA1 to 0xDF map to the half-width katakana characters found in JIS X 0201.

HTML written in Shift JIS can still be interpreted to some extent when incorrectly tagged as ASCII, and when the charset tag is in the top of the document itself, since the important start and end of HTML tags and fields, <, >, /, ", &, ; are coded by the same single bytes as in ASCII, and those bytes won't appear in two-byte sequences. Shift JIS is possible to use in string literals in programming languages such as C, but the 0x5C byte will cause problems when it appears as second byte of a two-byte character, because 0x5C, normally backslash, here ¥, will be interpreted as an escape sequence which will mess up the interpretation. If the programmer is aware of this, it would be possible to use printf("ハローワールド¥n"); (where ハローワールド is Hello, world and ¥n is an escape sequence), assuming the I/O system supports Shift JIS output.

Shift JIS requires an 8-bit clean medium for transmission. It is fully backwards compatible with the legacy JIS X 0201 single-byte encoding, meaning it supports half-width katakana and that any valid JIS X 0201 string is also a valid Shift JIS string. For two-byte characters, however, Shift JIS only guarantees that the first byte will be high bit set (0x80–0xFF); the value of the second byte can be either high or low. Appearance of byte values 0x40–0x7E as second bytes of code words makes reliable Shift JIS detection difficult, because same codes are used for ASCII characters. Since the same byte value can be either first or second byte, string searches are difficult, since simple searches can match the second byte of a character and the first byte of the next, which is not a real character. String search algorithms must be tailor made for Shift JIS.

On the other hand, the competing 8-bit format EUC-JP, which does not support single-byte halfwidth katakana, allows for a much cleaner and direct conversion to and from JIS X 0208 code points, as all high bit set bytes are parts of a double-byte character and all codes from ASCII range represent single-byte characters. Unicode also does not have some of the disadvantages of Shift JIS. Unicode does not have ambiguous versions; new characters are assigned to unused places by a single organisation. For Shift JIS, companies work in parallel. Unicode is backwards compatible with ASCII also for 0x5C, and does not have the string search problem.

For a double-byte JIS sequence $j_1 j_2$,^[2] the transformation to the corresponding Shift JIS bytes $s_1 s_2$ is:

$s_1 = \begin{cases} \left \lfloor \frac{j_1 + 1}{2} \right \rfloor + 112 & \mbox{if } 33 \le j_1 \le 94 \\ \left \lfloor \frac{j_1 + 1}{2} \right \rfloor + 176 & \mbox{if } 95 \le j_1 \le 126 \end{cases}$

$s_2 = \begin{cases} j_2 + 31 + \left \lfloor \frac{j_2}{96} \right \rfloor & \mbox{if } j_1 \mbox{ is odd }\\ j_2 + 126 & \mbox{if } j_1 \mbox{ is even } \end{cases}$

Multiple versions

Many different versions of Shift JIS exist.

There are two areas for expansion: Firstly, JIS X 0208 does not fill the whole 94×94 space encoded for it in Shift JIS, therefore there is room for more characters here — these are really extensions to JIS X 0208 rather than to Shift JIS itself. The most popular extension here is to the Windows-31J, otherwise known as Code page 932, popularized by Microsoft, although Microsoft itself does not recognize the Windows-31J name and instead calls that variation "shift_jis". Secondly, Shift JIS has more encoding space than is needed, for JIS X 0201 and JIS X 0208 and this space can and is used for yet more characters. The space, with lead bytes 0xF5 to 0xF9, is used by Japanese mobile phone operators for pictographs for use in E-mail, for example. (KDDI goes further and defines hundreds more in the space with lead bytes 0xF3 and 0xF4).

Beyond even this, there have been numerous minor variations made on Shift JIS, with individual characters here and there altered. Most of these extensions and variants have no IANA registration, so there is much scope for confusion, if the extensions are used. Microsoft Code Page 932 is registered separately from Shift JIS.

IBM CCSID 943 has the same extensions as Code Page 932. As with most code pages and encodings, it is recommended by Microsoft, Apple, the Unicode Consortium and most major operating system makers that Unicode be used instead.

Shift JIS byte map

The chart below gives the detailed meaning of each byte in a Shift JIS encoded stream.

First byte 0 1 2 3 4 5 6 7 8 9 A B C D E F 0 ␀ ␁ ␂ ␃ ␄ ␅ ␆ ␇ ␈ ␉ ␊ ␋ ␌ ␍ ␎ ␏ 1 ␐ ␑ ␒ ␓ ␔ ␕ ␖ ␗ ␘ ␙ ␚ ␛ ␜ ␝ ␞ ␟ 2 ␠ ! " # $ % & " ( ) * + , - . / 3 0 1 2 3 4 5 6 7 8 9  : ; < = > ? 4 @ A B C D E F G H I J K L M N O 5 P Q R S T U V W X Y Z [ ¥ ] ^ _ 6 ` a b c d e f g h i j k l m n o 7 p q r s t u v w x y z { | } ‾ ␡ 8 9 A ｡ ｢ ｣ ､ ･ ｦ ｧ ｨ ｩ ｪ ｫ ｬ ｭ ｮ ｯ B ｰ ｱ ｲ ｳ ｴ ｵ ｶ ｷ ｸ ｹ ｺ ｻ ｼ ｽ ｾ ｿ C ﾀ ﾁ ﾂ ﾃ ﾄ ﾅ ﾆ ﾇ ﾈ ﾉ ﾊ ﾋ ﾌ ﾍ ﾎ ﾏ D ﾐ ﾑ ﾒ ﾓ ﾔ ﾕ ﾖ ﾗ ﾘ ﾙ ﾚ ﾛ ﾜ ﾝ ﾞ ﾟ E F	Second byte 0 1 2 3 4 5 6 7 8 9 A B C D E F 0 1 2 3 4 5 6 7 8 9 A B C D E F
Non printable ASCII character Unaltered ASCII character Modified ASCII character Single-byte half-width katakana First byte of a double-byte JIS X 0208 character Unused as first byte of a JIS X 0208 character Second byte of a double-byte JIS X 0208 character whose first half of the JIS sequence was odd Second byte of a double-byte JIS X 0208 character whose first half of the JIS sequence was even Unused as second byte of a JIS X 0208 character

See also

• Japanese language and computers
• Mojibake
• Shift JIS art
• Microsoft code page 932

References

1. http://w3techs.com/technologies/history_overview/character_encoding
2. j₁ and j₂ are each in the range 33 (0x21) to 126 (0x7e) inclusive (i.e., 7-bit character values excluding control characters (0–31 (0x1f) and 127 (0x7f)) and space)

External links

• Shift-JIS A table of the non-ASCII part of the codeset.
• Microsoft's definition of Code Page 932
• Forms of Shift-JIS in ICU (International Components for Unicode)
  • ibm-942 (sjis78)
  • ibm-943 (Contains the \u00A5 ↔ \x5C mapping)
  • Shift JIS (Contains the \u005C ↔ \x5C mapping)