On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.

So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.

> UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.

Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.

Someone at the consortium is rubbing their hands with glee with all the newfound space.

But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?

UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(

Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.

Nobody needs more than 4.47 trillion characters. (famous last words)

> No special cases introduced. All properties preserved.

I don’t actually know if this is LLM-generated, but phrasing like this is weirdly triggering to me now

> Ken Thompson: "...i really dont think it is useful. it is like replacing ipv6 with ipv50"

At some point it just collapses into a sort of Huffman coding of every possible 4096 bit embedding vector.

FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.

On the naming issue with a "UTF-8" prefix, I suggest "UTF+8" which puns earlier in the string for a double effect. Fun paper!

Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?

Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.

Since the Unicode Consortium isn't going to actually assign those code points, this is functionally just a scheme for encoding variable-length integers designed as an extension of UTF-8 more or less arbitrarily.

There's a long history of designs for these (https://en.wikipedia.org/wiki/Variable-length_integer) that the author might be interested in. I used to think about these things myself, including the "zigzag encoding" for signed values (not a difficult idea; this "marvelous bijective mapping" is the standard one used in math class to demonstrate that the integers are countable, and the nice implementation properties are a consequence of the choice to "zig" from 0 to -1 first combined with how two's-complement works).

Some ideas of what to do with this space:

- fully-customizable emojis (think of a RPG-like character customization screen)

- heck, why not full jpegs/gifs?

- some unicode programming script (running Doom)

- ?

That said, some very minor (HN-style) nitpick:

> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.

Isn't a limit by definition no longer dependent on n?

IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.

I think at least 7 should be allowed. 42 bits could contain all sorts of information.

"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.

Why encode the length at all in the extension? Either the next byte is a continuation byte or not. Are there real world use cases where the length encoding is used in UTF-8 where individual bytes don’t have to be read anyway (for example, codepoint counting, but in practice what good is that without knowing if the code points are combining)? Even in the case where you just want a byte count, an invalid UTF-8 stream won’t respect the first count marker. The arbitrarily sized count is no better than an arbitrarily sized code point - you’d still have to handle all the same resource and sizing issues for safety. And then what, are you going to allocate a buffer that can fit a length that needs more than 64-bits to represent?

undefined

I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.

Alright, so now we can fit an entire LLM model into one character.

The letter from Ken Thompson is spot on:

"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.

i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."

On the opposite end of the spectrum from this idea, I've been thinking for a long time now that much of what's weird/redundant about Unicode comes from its self-synchronization requirement. And that you could create a very compact and flexible "re-embedding" of Unicode if you dropped this requirement.

Self-synchronization is the idea that if you take an arbitrary Unicode-encoding-encoded text stream, and then do any combination of 1. flipping bits in it at random, 2. injecting random extra whole bytes into the stream, and 3. dropping random whole bytes from the stream, then each such change will corrupt at most one Unicode code-unit in the stream, and from that, corrupt at most one semantic grapheme-cluster encoded by the stream. No such change will corrupt the stream itself so as to leave the stream in an invalid/indeterminate state that a Unicode parser can't know how to recover from. You'll have a one-character-wide "hole", and then the stream will resume. Unless that hole occurred at a character that's critical to the stream's meaning on an application-semantics level, the document will still be valid/useful (especially for archival/forensic-recovery purposes); just like a printed document is still valid/useful even if you drip a bit of ink on it.

Unicode encodings are self-synchronizing at the byte-pattern level. But, much less often discussed, Unicode itself is also designed to be self-synchronizing in how it encodes interactions between code-units. (In other words, Unicode is designed to never have modal or stateful semantics, beyond the boundary of a single grapheme cluster.) And this really constrains how certain Unicode features can be, and historically have been, designed.

If you want a run of Unicode code-units to all be "tagged" or "colored" with some property, then, due to the self-synchronization requirement (i.e. due to the assumption that any single one of those bytes could be corrupted or blown away, including whatever metadata-encoding bytes your scheme wants to use), you have to either:

- define a "pre-colored" alphabet, and express your tagged content in that alphabet. (Think of e.g. the Unicode flag codepoints [https://en.wikipedia.org/wiki/Regional_indicator_symbol], which are essentially a special namespaced copy of the roman upper-case alphabet intended to be used only to spell out two-letter contiguous pairs that are [or at some point were] valid ISO country codes; where, when used in this way, the resulting 'colored'-letter-pair sequence has 'flag semantics', i.e. is meant to be rendered as a flag and machine-legible as a flag)

- or individually tag each and every one of those codepoints with its own tag/color codepoint (as in Unicode variation selectors)

- or interject binary-infix "operator" codepoints as glue between each codepoint in a codepoint sequence, so that those "operator" codepoints each affix together their immediate sibling codepoints, essentially constructing an abstract-Unicode-semantics list ADT "cons by cons", so that said list then may then be assigned its own semantics, e.g. being treated as a single grapheme-cluster with its own rendering (as in e.g. ZERO WIDTH JOINER used in its role in constructing complex emoji)

These encodings are all very high-overhead; but these are the kinds of trade-offs you have to make for self-synchronization to work.

And these trade-offs are sensible to make... if you're Ken Thompson in 1992, having to consider e.g. plaintexts being transmitted over raw RS232, or filesystems that just blast bytes to a spinning-rust disk without so much as a checksum, and then read them back "blindly" years later with that disk potentially highly-degraded.

But what if you live in the modern world, and you only care about holding and manipulating known-length strings in memory and/or embedded into code-signed (and thereby hashed) binaries; checksummed-block filesystems over rarely-corrupting NVMe; and transmission of data mostly over encrypted-stream protocols, where even the rare packet-level corruption that still TCP-checksums correctly, doesn't decrypt successfully, and therefore causes TLS-level retransmission?

Well, then you could define a much-more-concise reformulation of Unicode, that uses all the "forbidden" semantics-encoding techniques that Unicode itself avoids due to the self-synchronization constraint.

Where by "reformulation", I mean: a standard that keeps parity with Unicode in terms of what it can encode; and which at all times maintains a clearly-defined lossless bijective transformation between it and Unicode, evolving in lockstep with Unicode; but where Unicode and this formulation have their own distinct universes of codepoints, that compose using different rules, into the same ultimate sets of reachable grapheme-clusters with the same text-segmentation/collation/etc semantics.

I'm honestly kind of surprised that there isn't already a project somewhere to define an alternative Unicode formulation that looks like this.

It'd not only be potentially a highly-efficient representation for many in-memory string operations (that text shaping libraries would also love); it'd also likely perform impressively (compared to regular UTF-8) as a canonical representation for documents used to train+prompt LLMs. It'd give "more meaning per token", via all the repeated-per-codel overhead becoming once-per-sequence overhead; and it'd also allow many layers of meaning that are currently encoded via in-band protocols (ANSI escape codes, Markdown, HTML/XML, etc) that the LLM needs to learn additional recognition logic for, to instead be parsed out "during" initial text-stream recognition.

(And it's taking me real willpower not to go into depth on all the features such a formulation could have, and all the benefits it could provide. I should probably stop here before I nerd-snipe myself!)

I love it.

Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.

Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)

Logo and name need more work but other than that, ship it.

This is like TIME CUBE (https://en.wikipedia.org/wiki/Time_Cube) but for encoding nerds.

edit: replaced false prophet site with wiki link for the original one and true timecube.com...lost to the sands of ...time.

Finally, an ASCII-safe way to encode all the vectors of Neuralese.

More like WTF-8.

https://xkcd.com/1953/

I have entertained ideas like this every since becoming familiar with the original RFC2279 UTF-8 well over two decades ago, before UTF-8 was subsequently restricted to 21 bits.

Another idea that would work is to start an encoding with 0b10xxxxxx have subsequent 0b10xxxxxx bytes continue the encoding, and end it with 11xxxxxx. 0b1000_0000 (i.e. 128) is not allowed for the first byte of this sequence, and we add 127 to the resulting number (so each code point has only one representation) This way, the ♥ symbol (that’s 0x2665 or 9829 in decimal) would be represented as follows:

♥ → 0b0010_0110_0110_0101 → 0b0010_0101_1110_0110 (subtract 127) → 0b10_00_0010 0b10_0101_11 0b11_10_0110

This allows 18 bits to be encoded in three bytes, and, like UTF-8000, allows arbitrarily long sequences.

In the real world, the last time I needed to have a custom encoding, I went the other way and converted Unicode in to a 7-bit ASCII compatible encoding, where most ASCII control characters were converted into glyphs, as follows:

    0123456789abcdef
  0 .ÁÉÍÑÓÚÜ¡..—..«»
  1 •áéíñóúü¿‘’.→“”©
  2 .!"#$%&'()*+,-./
  3 0123456789:;<=>?
  4 @ABCDEFGHIJKLMNO
  5 PQRSTUVWXYZ[\]^_
  6 `abcdefghijklmno
  7 pqrstuvwxyz{|}~♥

The reason for this encoding is that it allows me to write stuff in languages I am actually fluent in (English and Spanish), and adds a handful of useful non-ASCII punctuation (smart quotes, etc.).

“.” represents a control character here; only “null” (0x00), “line feed” (0x0a), “form feed” (0x0c), “tab” (0x09), “carriage return” (0x0d), and “escape” (0x1b) are control characters; the rest are printable glyphs. I used this character set for my blog, since my blog is processed using HTMLDOC (which I learned the hard way is buggy with full Unicode) and my own Lua script.

With Lua, Unicode regexes are not readily supported because its regular expression engine assumes a codepoint is only one byte long when forming character classes such as %u for upper case letters. My Lua code has character classes for the non-ASCII glyphs such as %t for all letters in this custom encoding.

I have made this encoding 7 bits because that gives me the option to use the eight bit for future expansion.

The real reason for so few glyphs is because, in addition to having things work nicely with code which assumes 8-bit codepoints, with the modern web, I need to send to the viewer the font a page will be rendered with (it would had been nice if Microsoft had open sourced Verdana, Georgia, and the rest of the core fonts, and had those fonts became a part of the browser standards, so we could have cross-platform font stacks, but oh well) [1], so I aggressively subset the font to minimize the page load time.

[1] I still wish for the day when Apple, Google, Microsoft, and Mozilla come together, decide to include “Noto” and “Noto Serif” with their browsers, so “font-family: Noto (Serif)” always renders the exact same font without having to download a font over the network when loading a web page.

It would be nice if the first paragraph gave a motivation for this. Why did you start this?

I mean, I can derive from the text that this can encode arbitrarily large integers. So what? What's the point? I also understand that this is for fun.

No project is ever safe from complicators.

This is why we need the KISS enforcers.

Perhaps we can use a generalized Unicode-like encoding space for LLM text tokens. A text tokenization scheme uses up a few hundred thousand entries, with say one thousand new entries added annually. These can be called amojis, meaning AI mojis.