Re: Internal Representation of Strings

Marco van de Voort schrieb:


>>> in general, UTF-8 takes less space than UTF-16 (and mixes much better with
>>> code designed for ASCII), but some many languages like UTF-16 more
>>> potentially because it works better when being treated as an array.
>> This IMO is a typical misconception of English-only speakers, which have
>> caused a lot of trouble in the evolution of programming languages :-(
>
> Latin only? But afaik even for Cyrillic and the Semitic language group it
> doesn't matter.


The standard libraries were restricted to the ASCII character set in
the past, with trouble in all languages with more or entirely
different characters. The introduction of codepages then allowed to
use at least 256 characters, and nowadays most people extend the frame
only to the Unicode BMP, and ignore other codepages with regards to
memory and runtime requirements.


Unicode itself was a big leap, but introduced new problems. The UCS-2
standard ignored (willingly?) the amount of characters in well known
Chinese "character" sets. Tell me a programming language with proper
support for native language text in string literals, and Unicode
source code representation. The C extension with wide string literals
(L"...") is a bad hack, because it ignores the existence of various
possible character encodings. It had been easier to stay with 8 bit
characters, and only move to UTF-8 as the unique encoding for strings,
so that every Unicode-aware editor could be used to edit source code
in any natural language, and could store everything in a unique UTF-8
source file encoding.


Not much better outside programming languages, also HTML and related
standards use inconsistent encodings, differently restricted
characters sets in dedicated places, and different escape sequences.


DoDi

Return to the comp.compilers page.
Search the comp.compilers archives again.