Skip to content

ENH: Convert tanh from C universal intrinsics to C++ using Highway #25934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mousius merged 10 commits into from
Dec 4, 2024
Merged

ENH: Convert tanh from C universal intrinsics to C++ using Highway #25934

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
73975f8
ENH: Convert tanh from C universal intrinsics to C++ using Highway
sterrettm2 Feb 14, 2024
260200e
Now uses 4-way lookup table for AVX2
sterrettm2 Mar 6, 2024
6286a3b
Fixed some #if guards and other minor changes
sterrettm2 Mar 6, 2024
eb29a74
Moved more things into #if guards and cleaned up implementation
sterrettm2 Mar 7, 2024
41e6446
Remove VX due to lack of support, minor cleanup
sterrettm2 Nov 7, 2024
43d9097
Use Highway's detection for native FMA
sterrettm2 Nov 12, 2024
0ae0be0
Remove duplication and minor cleanup
sterrettm2 Nov 21, 2024
16b11fc
Changed transposed LUT logic to use constexpr if instead of idefs
sterrettm2 Nov 22, 2024
9d3c046
Clean up load types to fix build errors
sterrettm2 Nov 22, 2024
099c44e
Update highway
sterrettm2 Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Now uses 4-way lookup table for AVX2
  • Loading branch information
@sterrettm2
sterrettm2 committed Dec 2, 2024
commit 260200e3882d77d3dafa62bf8623d21699f08494
26 changes: 26 additions & 0 deletions numpy/_core/src/umath/loops_hyperbolic.dispatch.cpp.src
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,19 @@ HWY_ATTR NPY_FINLINE vec_f64 lut_16_f64(const double * lut, vec_u64 idx){
const vec_f64 lut0 = hn::Load(f64, lut);
const vec_f64 lut1 = hn::Load(f64, lut + 8);
return hn::TwoTablesLookupLanes(f64, lut0, lut1, hn::IndicesFromVec(f64, idx));

@jan-wassenberg jan-wassenberg Mar 6, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might actually be worthwhile to implement an AVX2 version with 4 table lookups and blending, because gather is quite slow again.

}else if constexpr (hn::Lanes(f64) == 4){
const vec_f64 lut0 = hn::Load(f64, lut);
const vec_f64 lut1 = hn::Load(f64, lut + 4);
const vec_f64 lut2 = hn::Load(f64, lut + 8);
const vec_f64 lut3 = hn::Load(f64, lut + 12);

const auto high_mask = hn::Ne(hn::ShiftRight<3>(idx), hn::Zero(u64));
const auto load_mask = hn::And(idx, hn::Set(u64, 0b111));

const vec_f64 lut_low = hn::TwoTablesLookupLanes(f64, lut0, lut1, hn::IndicesFromVec(f64, load_mask));
const vec_f64 lut_high = hn::TwoTablesLookupLanes(f64, lut2, lut3, hn::IndicesFromVec(f64, load_mask));

return hn::IfThenElse(hn::RebindMask(f64, high_mask), lut_high, lut_low);
}else{
return hn::GatherIndex(f64, lut, hn::BitCast(s64, idx));
}
Expand All @@ -121,6 +134,19 @@ HWY_ATTR NPY_FINLINE vec_f32 lut_32_f32(const float * lut, vec_u32 idx){
const vec_f32 lut0 = hn::Load(f32, lut);
const vec_f32 lut1 = hn::Load(f32, lut + 16);
return hn::TwoTablesLookupLanes(f32, lut0, lut1, hn::IndicesFromVec(f32, idx));
}else if constexpr (hn::Lanes(f32) == 8){
const vec_f32 lut0 = hn::Load(f32, lut);
const vec_f32 lut1 = hn::Load(f32, lut + 8);
const vec_f32 lut2 = hn::Load(f32, lut + 16);
const vec_f32 lut3 = hn::Load(f32, lut + 24);

const auto high_mask = hn::Ne(hn::ShiftRight<4>(idx), hn::Zero(u32));
const auto load_mask = hn::And(idx, hn::Set(u32, 0b1111));

const vec_f32 lut_low = hn::TwoTablesLookupLanes(f32, lut0, lut1, hn::IndicesFromVec(f32, load_mask));
const vec_f32 lut_high = hn::TwoTablesLookupLanes(f32, lut2, lut3, hn::IndicesFromVec(f32, load_mask));

return hn::IfThenElse(hn::RebindMask(f32, high_mask), lut_high, lut_low);
}else{
return hn::GatherIndex(f32, lut, hn::BitCast(s32, idx));
}
Expand Down