Missing UTF-8 decode that passes through BOM but replaces surrogates

[domenic]

I'm trying to use utf8.js in jsdom in preference to native TextDecoder()/TextEncoder() because of the correctness issues you've fixed in this library.

However, I don't think it has everything I need.

• Spec UTF-8 decode -> skip BOM, replace surrogates
• Spec UTF-8 decode without BOM -> pass through BOM, replace surrogates
• Spec UTF-8 decode without BOM or fail -> pass through BOM, throw on surrogates
• Spec UTF-8 encode -> there are no variants, because it assumes the input is always a scalar value string

Whereas:

• utf8fromString vs. utf8fromStringLoose should always be equivalent because input is always a scalar value string.
• utf8toString: decode, pass through BOM, throw on surrogates
• utf8toStringLoose: decode, skip BOM, replace surrogates

So I believe I'm missing a "decode, pass through BOM, replace surrogates" operation. Would you be up for providing it?

(I also find the terminology of "loose" vs. not quite confusing, especially since it seems to mean different things on encode vs. decode... Just giving clear documentation in the README would help here.)

[domenic]

• utf8toStringLoose: decode, skip BOM, replace surrogates

Oh, this appears to not skip the BOM actually? Despite my reading of the code.

const { utf8toStringLoose } = require('@exodus/bytes/utf8.js');

// UTF-8 BOM + {"key":"value"}
const withBom = new Uint8Array([0xEF, 0xBB, 0xBF, 0x7B, 0x22, 0x6B, 0x65, 0x79, 0x22, 0x3A, 0x22, 0x76, 0x61, 0x6C, 0x75, 0x65, 0x22, 0x7D]);

const result = utf8toStringLoose(withBom);
console.log('Result:', result);
console.log('First char code:', result.charCodeAt(0));
console.log('Starts with BOM:', result.charCodeAt(0) === 0xFEFF);

try {
  const parsed = JSON.parse(result);
  console.log('Parsed:', parsed);
} catch (e) {
  console.log('Parse error:', e.message);
}

Prints

Result: {"key":"value"}
First char code: 65279
Starts with BOM: true
Parse error: Unexpected token '', "{"key":"value"}" is not valid JSON

In that case I guess what'd be missing is something which skips the BOM and replaces surrogates.

[ChALkeR]

utf8fromString vs. utf8fromStringLoose should always be equivalent

Only for jsdom/url/encoding spec.
In general, JS strings can contain unpaired surrogates, and that's where the difference is between those two.

/utf8.js (and /utf16.js) is not BOM-aware, similar to Buffer.from/Buffer.#toString API.
/encoding.js is BOM-aware and has a helper method for BOM detection and option to control BOM behavior in the decoder.

utf8toString: decode, pass through BOM, throw on surrogates
utf8toStringLoose: decode, skip BOM, replace surrogates

What's passthough/skip?

The behavior on BOM should be identical for loose and non-loose methods:

> require('@exodus/bytes/utf8.js').utf8fromString('\uFEFF')
Uint8Array(3) [ 239, 187, 191 ]
> escape(require('@exodus/bytes/utf8.js').utf8toString(_))
'%uFEFF'
> require('@exodus/bytes/utf8.js').utf8fromStringLoose('\uFEFF')
Uint8Array(3) [ 239, 187, 191 ]
> escape(require('@exodus/bytes/utf8.js').utf8toStringLoose(_))
'%uFEFF'

Assuming passthrough is keeping BOM and skip means removing BOM (if it's the first codepoint), then

utf8toStringLoose: decode, skip BOM, replace surrogates

is incorrect, if it skips BOM on any input that is a bug

[ChALkeR]

Ah, just noticed the comment!

In that case I guess what'd be missing is something which skips the BOM and replaces surrogates.

How is that different from:

import { TextDecoder } from `@exodus/bytes/encoding-lite.js` // or encoding.js if you are already loading it
new TextDecoder().decode(bytes)

Alternatively:

import { getBOMEncoding } from `@exodus/bytes/encoding-lite.js`
import { utf8toStringLoose } from '@exodus/bytes/utf8.js'
utf8toStringLoose(getBOMEncoding(u8) === 'utf-8' ? u8.subarray(3) : u8)

[ChALkeR]

I'm trying to use utf8.js in jsdom in preference to native TextDecoder()/TextEncoder() because of the correctness issues you've fixed in this library.

Also, for utf-8 decoding, new TextDecoder().decode() is fine everywhere where it's present and is native.

utf-8 TextDecoder is bugged in these cases:

1. A non-native impl/polyfill using Buffer or iconv-lite, bundled for non-Node.js platforms, e.g. React Native or browsers (where it could be used directly despite native impl being present)
2. Chrome, Safari and Bun in { stream: true } mode
3. Firefox if you reuse TextDecoder instance, i.e. const decoder = new TextDecoder(); decoder.decode() instead of new TextDecoder().decode() on every call

Nether of those are jsdom targets? (if it's not using streaming)
Most likely, using TextDecoder should be just fine there

See overall table for more data: https://docs.google.com/spreadsheets/d/1pdEefRG6r9fZy61WHGz0TKSt8cO4ISWqlpBN5KntIvQ/edit?gid=672941037#gid=672941037

[domenic]

Only for jsdom/url/encoding spec.
In general, JS strings can contain unpaired surrogates, and that's where the difference is between those two.

Agreed and understood.

What's passthough/skip?

passthrough is the behavior of the TextDecoder APIs with ignoreBOM: true: BOM bytes decode to U+FEFF. This corresponds to the Encoding Standard's "UTF-8 decode without BOM (or fail)" algorithms.

skip is the default behavior of the TextDecoder APIs, or the Encoding Standard's "UTF-8 decode" algorithm: discard the first three bytes if they match the BOM.

if it skips BOM on any input that is a bug

Understood. I got confused by

bytes/utf8.js

Lines 10 to 12 in 0102316

	const decoderLoose = nativeDecoder
	const decoderFatal = nativeDecoder
	? new TextDecoder('utf-8', { ignoreBOM: true, fatal: true })

. I thought decoderLoose would leave the default of ignoreBOM: false, but I failed to follow the actual definition of decoderLoose in the other file, where I see it actually ignoreBOM: true.

How is that different from:

Also, for utf-8 decoding, new TextDecoder().decode() is fine everywhere where it's present and is native.

I agree it's possible to use other parts of the @exodus/bytes public API, or the native platform APIs, to accomplish what I need. I was just hopeful that I could directly use @exodus/bytes/utf8.js from my call sites, as simple ways to directly accomplish the spec operations, without creating ambient TextDecoder (or TextEncoder) objects.

In particular, it would allow a simple mental model of "always use @exodus/bytes, never the native platform APIs". This is nice because I know I can trust @exodus/bytes to be the most correct and performant option, whereas recent events have made me doubt the native platform APIs.

That said, I can understand if providing beautiful APIs for people doing jsdom-style web spec implementations is not a goal of this library. It makes sense to instead just supply the base functionality in whatever shape you think is best for normal users, and require jsdom to do the extra work of deciding between utf8toString, utf8toStringLoose, or TextDecoder.

I think I'll add a small wrapper layer to jsdom that exposes appropriate @exodus/bytes operations (or native platform operations) with their spec-aligned names, and use that throughout the codebase instead of using @exodus/bytes directly.

[domenic]

Here is where I ended up: jsdom/jsdom@1648a8a

My suggestion is that you close this issue by clearly documenting utf8.js's four functions with regard to exactly what they do in regard to both BOMs and lone surrogates, similar to the jsdoc I added in my own commit.

Optionally, you could link to the corresponding spec algorithms, but that's probably not of interest to most consumers. Maybe it's more obvious to just tell people the corresponding options for TextEncoder/TextDecoder for both the defaults and the loose versions. (Including noting that there is no TextEncoder version corresponding to utf8fromString(), since the deLoose() pass for that function goes and seeks out any replacement characters in a fairly expensive way.)

[ChALkeR]

Thanks! Yes, documentation is currently lacking for many of the methods and is on the roadmap