Dealing with LSP Encoding
Microsoft's Language Server Protocol requires UTF-8 encoding for your text, but by default it wants you to measure offsets and positions using UTF-16 "code units", which is a different encoding system. This mismatch makes it difficult to comply with the specification. (Unless you happen to be writing in Java or JS or C#, which all match the strange UTF-16 "code unit" behavior by default)
How to opt out of the Code Units
Your first goal should be to try to negotiate for UTF-8 byte offsets instead of UTF-16 code units.
The LSP capability negotiation works like this:
- Client sends `initialize` request: The client lists which position encodings it supports, in order of preference. To negotiate UTF-8 as a client, put
utf-8first:
{"capabilities": { "general": {"positionEncodings": ["utf-8", "utf-16"] }}}
- Server responds: The server should pick
utf-8from the client's list if it supports it:
{"capabilities": {"positionEncoding": "utf-8"}}- If the server omits
positionEncoding, the negotiation falls back toutf-16, even if the client didn't put"utf-16"in its list.
- Both sides use the agreed encoding: Once negotiated, all position and offset measurements in the protocol use this encoding for the rest of the session.
When the server chooses utf-8, both sides can measure positions in plain byte offsets.
How to deal with the Code Units
If the above negotiation fails, then you have to deal with UTF-16 code units.
A code point is Unicode's term for a single character in its character set. UTF-8 uses between 1 and 4 bytes per code point:
| Code point | a | α | ａ | 𝕒 |
|---|---|---|---|---|
| # of bytes | 1 | 2 | 3 | 4 |
| # of codepoints | 1 | 1 | 1 | 1 |
| # of UTF-16 code units | 1 | 1 | 1 | 2 |
You can see that UTF-16 code units are different from counting codepoints. Your Unicode library probably supports codepoints, but for LSP compliance, you need code units.
Instead of incorrectly using codepoints from a library, I recommend computing the code unit offsets yourself. This involves iterating over your string and counting a running total.
One way to count the number of UTF-16 code units is to iterate through each code point in your string and refer to the table above, counting each code point as +1, except 4-byte code points, which are +2.
Byte-level counting trick
Another approach is to iterate over raw bytes instead of code points. In some languages, iterating over bytes is simpler or faster than decoding UTF-8 code points, especially if your standard library doesn't provide good UTF-8 iteration.
The key insight is that UTF-8's byte structure tells us how many bytes are in each code point:
- Bytes
0-127(0xxxxxxx): Start of a 1-byte code point - Bytes
128-191(10xxxxxx): Continuation byte (not a code point start) - Bytes
192-223(110xxxxx): Start of a 2-byte code point - Bytes
224-239(1110xxxx): Start of a 3-byte code point - Bytes
240-247(11110xxx): Start of a 4-byte code point
Since UTF-16 code units count 1-3 byte code points as 1 unit and 4-byte code points as 2 units, we can count as we iterate over bytes:
- Bytes
0-191: Add +1 (either a complete 1-byte code point, or a continuation byte) - Bytes
192-223: Add +0 (start of a 2-byte code point, canceling out the continuation byte) - Bytes
224-247: Add -1 (start of a 3-byte or 4-byte code point, canceling out 2-3 continuation bytes)
If you're doing this byte trick, the continuation byte being +1 is important: this ensures that when you reach a target offset, you're positioned at the end of a code point, not in the middle of one.