Your model returned start 43, end 49. Four systems read that as four different substrings. This page shows you which one your text actually produces.
The Language Server Protocol had to negotiate this. LSP 3.17 added a positionEncoding capability so a client and a server can agree on what a character offset counts, because they were not agreeing. From the specification:
Prior to 3.17 the offsets were always based on a UTF-16 string representation.
Since 3.17 clients and servers can agree on a different string encoding representation (e.g. UTF-8).
To stay backwards compatible the only mandatory encoding is UTF-16 represented via the string utf-16.
Implementation considerations: since the conversion from one encoding into another requires the content of the file / line the conversion is best done where the file is read which is usually on the server side.
Language Server Protocol Specification 3.17, Text Documents / Position. microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocuments
Every major editor implements LSP. The protocol still needed a negotiated capability, a backward-compatibility clause and a note about where the conversion belongs, all for one integer. Your model, your tokenizer, your redactor and your highlighter negotiate nothing.
Exactly the string the offsets were computed against. Trailing newlines count.
A textarea cannot hand this page a carriage return. The HTML standard: "The algorithm for obtaining the element's API value is to return the element's raw value, with newlines normalized." (WHATWG HTML, the textarea element), and that normalization makes every line break a lone LF. Tick the box if the system that produced your offsets counted CRLF, and every line ending is measured as two code units again.
Reads 43,49, 43 49, [43, 49], {"start":43,"end":49}, or a whole JSON array. A third field becomes a label. End is exclusive.
Every line on this page is a measurement of the text and the spans in the boxes above. None of it is a prediction about your production system.
Some things move every offset after them and raise nothing. Each one below is reported with the offset where the shift begins and the measured length change, rather than a bare note that two lengths differ.
| Convention | Length of this text | Systems that count this way |
|---|
JavaScript strings, DOM Range and CharacterData offsets, the LSP default utf-16, Java, .NET, Windows APIs. Java documents it plainly: "Index values refer to char code units, so a supplementary character uses two positions in a String." (java.lang.String, Java SE 21 API documentation).
Python 3 str, Go runes, LSP utf-32, and any counter that thinks in characters. "Strings are immutable sequences of Unicode code points." (Python documentation, Text Sequence Type - str). LSP records the equivalence: "Implementation note: these are the same as Unicode code points, so this PositionEncodingKind may also be used for an encoding-agnostic representation of character offsets."
Rust str, byte-level tokenizers, protobuf string fields, regex run over bytes, LSP utf-8. Rust refuses instead of mangling: "Panics if begin or end does not point to the starting byte offset of a character (as defined by is_char_boundary), if begin > end, or if end > len." (std::str). Protocol Buffers: "A string must always contain UTF-8 encoded or 7-bit ASCII text" (Language Guide proto3, Scalar Value Types).
What a reader sees, and what a redaction rectangle has to cover. Unicode Standard Annex #29 "describes guidelines for determining default boundaries between certain significant text elements: user-perceived characters, words, and sentences" and notes "The default grapheme clusters are also known as extended grapheme clusters." ECMA-402 lists UAX #29 as a normative reference, which is what Intl.Segmenter with granularity: "grapheme" segments on.
If Intl.Segmenter is missing, the grapheme column computes nothing and says so. There is no regex fallback here, because a regex approximation of grapheme clusters is wrong for exactly the ZWJ sequences, flag pairs and skin-tone modifiers this column exists for, and shipping it would reproduce the bug class this page is built to find. Intl.Segmenter reached Baseline newly available on 2024-04-16, supported from Chrome 87, Edge 87, Firefox 125, Safari 14.1 and Safari on iOS 14.5.