Skip to content

util: add fast path for Latin1 decoding #55275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nodejs-github-bot merged 15 commits into from
Dec 3, 2024
Merged

util: add fast path for Latin1 decoding #55275

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b12ba58
util: add fast path for Latin1 decoding
Oct 5, 2024
3c239bd
handle input correctly in decode functions
Oct 5, 2024
17f317e
repair
Oct 5, 2024
e0a40c6
Update lib/internal/encoding.js
mertcanaltin Oct 5, 2024
b7420b5
add ignoreBOM and fatal support to decodeLatin1
Oct 5, 2024
8bc747e
lint
Oct 5, 2024
fced029
update tests to support ignoreBOM and fatal for decodeLatin1
Oct 5, 2024
af7cca5
lint
Oct 5, 2024
659f1b6
correct BOM check in decodeLatin1 for uint8_t comparison
Oct 5, 2024
b92e571
add BOM test case to cover 0xFF in decodeLatin1
Oct 5, 2024
94109bf
Update encoding_binding.cc
mertcanaltin Oct 6, 2024
6dec37f
delete result.resize
Oct 6, 2024
f62f4c4
Update encoding_binding.cc
mertcanaltin Oct 7, 2024
f96e2ef
Update encoding_binding.cc
mertcanaltin Oct 7, 2024
77bc5ee
update text-decoder to use windows-1252 encoding
Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add ignoreBOM and fatal support to decodeLatin1
  • Loading branch information
Mert Can Altin
Mert Can Altin committed Oct 5, 2024
commit b7420b55210a745f68bef130fc707d28ff8e3eee
3 changes: 2 additions & 1 deletion lib/internal/encoding.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,13 +441,14 @@ function makeTextDecoderICU() {
validateDecoder(this);

this[kUTF8FastPath] &&= !(options?.stream);
this[kLatin1FastPath] &&= !(options?.stream);

if (this[kUTF8FastPath]) {
return decodeUTF8(input, this[kIgnoreBOM], this[kFatal]);
}

if (this[kLatin1FastPath]) {
Comment thread
anonrig marked this conversation as resolved.
return decodeLatin1(input);
return decodeLatin1(input, this[kIgnoreBOM], this[kFatal]);
}

this.#prepareConverter();
Expand Down
10 changes: 9 additions & 1 deletion src/encoding_binding.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,10 +260,18 @@ void BindingData::DecodeLatin1(const FunctionCallbackInfo<Value>& args) {
"SharedArrayBuffer, or ArrayBufferView.");
}

bool ignore_bom = args[1]->IsTrue();
bool has_fatal = args[2]->IsTrue();

ArrayBufferViewContents<uint8_t> buffer(args[0]);
const uint8_t* data = buffer.data();
size_t length = buffer.length();

if (ignore_bom && length > 0 && data[0] == 0xFEFF) {
data++;
length--;
}

if (length == 0) {
return args.GetReturnValue().SetEmptyString();
}
Expand All @@ -273,7 +281,7 @@ void BindingData::DecodeLatin1(const FunctionCallbackInfo<Value>& args) {
size_t written = simdutf::convert_latin1_to_utf8(
reinterpret_cast<const char*>(data), length, &result[0]);
Comment thread
mertcanaltin marked this conversation as resolved.
Outdated

if (written == 0) {
if (has_fatal && written == 0) {
return node::THROW_ERR_ENCODING_INVALID_ENCODED_DATA(
env->isolate(), "The encoded data was not valid for encoding latin1");
}
Expand Down