UTS #6: A Standard Compression Scheme for Unicode (SCSU).Ī: A Unicode transformation format (UTF) is anĪlgorithmic mapping from every Unicode code point (except surrogate code There are compression transformations such as the one described in the Unicode data, including UTF-8, UTF-16 and UTF-32. Q: Can Unicode text be represented in more than one way?Ī: Yes, there are several possible representations of One or two 16-bit code units, or a single 32-bit code unit. Depending on theĮncoding form you choose (UTF-8, UTF-16, or UTF-32), each character will then be represented either as a sequence of one to four 8-bit bytes, The Unicode Standard encodes characters in the range U+0000.U+10FFFF, which amounts to a 21-bit code space. The first version of Unicode was a 16-bit encoding, from 1991 to 1995, but starting with Unicode 2.0 (July, 1996), it has notīeen a 16-bit encoding. General questions, relating to UTF or Encoding FormĪ: No. Why wouldn’t I always use a protocol that requires a BOM?.How do I tag data that does not interpret U+FEFF as a BOM?.I am using a protocol that has BOM at the start of text.What should I do with U+FEFF in the middle of a file?.Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, does it affect the byte order?.When a BOM is used, is it only in 16-bit Unicode text?.How do I convert an unpaired UTF-16 surrogate to UTF-32?.How do I convert a UTF-16 surrogate pair such as to UTF-32? As one or as twoĔ-byte sequences?.Are there exceptions to the rule of exclusively using string parameters in APIs?.Doesn’t it cause a problem to have UTF-16 string APIs, instead of UTF-32 char APIs?.How about using UTF-32 interfaces in my APIs?.Should I use UTF-32 (or UCS-4) for storing Unicode strings in memory?.What is the difference between UCS-2 and UTF-16?.How should I handle supplementary characters in my code?.Because most supplementary characters are uncommon, does that mean I can ignore them?.
Can Unicode text be represented in more than one way?.This function converts from UTF-16 to codepoint firstly, then from codepoint to UTF-8.General questions, relating to UTF or Encoding Forms I have created a function that converts from UTF-16 to UTF-8.