Skip to content

GHA/http3-linux: add CPU and revision number to cache keys, switch 2 jobs to arm64 - #23214

Closed
vszakats wants to merge 12 commits into
curl:masterfrom
vszakats:gha-localbuild-dir
Closed

vszakats wants to merge 12 commits into
curl:masterfrom
vszakats:gha-localbuild-dir

Conversation

@vszakats

@vszakats vszakats commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Also move all local build installs under the ~/pkg/ root, replacing
~/<dep>/build/ and syncing with GHA/linux. Except quiche, which is a
Rust project and lacks (or I could not find) a way to do a classic
'install' to a custom prefix.

  • include the CPU architecture in the cache key name. To allow cache
    blobs adapt automatically to the runner machine arch. It allows to
    transparently use the arm runner machines for any jobs.
  • add a revision number ('1') to the cache keys and IDs.
    To make it easy to bump when doing modifications like this, or just
    want to make test that need rebuilding them.
  • merge two stray quiche verbose build flags into -vv.
    (Useful to see BoringSSL build progress)
  • tried moving quiche from the default target subdir into the ~/pkg/
    root. It failed because quiche headers remain inside the source tree,
    ~/quiche/quiche/include, and missing from the cache.
  • build all cached dependencies for arm64 as well.
  • switch BoringSSL jobs to arm64.

Note:

  • quiche has a valgrind issue, OpenSSL 3 and 4 another one,
    when built on arm64. Not tackled in this PR.

Datapoint: Rebuilding all cached dependencies takes:

In the actual jobs, arm64 is slightly faster in places, similar in most
others:
x64: https://github.com/curl/curl/actions/runs/37019925832
arm64: https://github.com/curl/curl/actions/runs/37026860992

Possible minor downside:

  • actual jobs now need to wait for both arm64 and intel cache builds
    to complete if any of the dependencies change. This happens rarely. It
    may also take a little bit longer to verify if the caches need to be
    rebuilt.

Follow-up to 9fd900c #23233


perf x64 vs arm64:

@vszakats
vszakats marked this pull request as draft October 1, 2026 11:35
@vszakats

This comment was marked as resolved.

@github-actions github-actions Bot added the CI Continuous Integration label Oct 1, 2026
vszakats added a commit that referenced this pull request Oct 1, 2026
@vszakats
vszakats force-pushed the gha-localbuild-dir branch 7 times, most recently from f9f26e6 to 7b3bae2 Compare October 1, 2026 17:57
@vszakats vszakats changed the title GHA: try syncing up local install directories GHA/http3-linux, linux: move and sync up install directories of local builds Oct 1, 2026
@vszakats
vszakats marked this pull request as ready for review October 1, 2026 18:03
@vszakats
vszakats requested a lite review from Copilot October 1, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Resolve the outstanding HTTP/3 workflow issues before approval.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Moves local dependency builds under ~/pkg/ and revisions cache identifiers across Linux and HTTP/3 workflows.

Changes:

  • Synchronizes dependency, cache, and configuration paths.
  • Adds cache revision 1.
  • Updates HTTP/3 build references.
File Summary
.github/​workflows/​linux.yml Updates Linux dependency paths and cache identifiers.
.github/​workflows/​http3-linux.yml Updates HTTP/3 paths and caches; duplicate quiche verbosity flags and stale quiche path comments remain to be corrected.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/http3-linux.yml
Comment thread .github/workflows/http3-linux.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The HTTP/3 workflow references an incorrect nghttp2 pkg-config directory.

Review effort: Lite
Findings: None

Resolved since last review (2)

vszakats added a commit to vszakats/curl that referenced this pull request Oct 1, 2026
vszakats added a commit that referenced this pull request Oct 1, 2026
Also sync option order in CMake sibling.

Possibly hinted by Copilot
Bug: #23214 (review)

Closes #23226
@vszakats
vszakats force-pushed the gha-localbuild-dir branch 2 times, most recently from a05abfa to 15fff50 Compare October 1, 2026 19:30
@vszakats
vszakats requested a lite review from Copilot October 1, 2026 19:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The Linux workflow still installs and caches rustls under ~/rustls instead of ~/pkg/rustls.

Review effort: Lite
Findings: None

@vszakats

vszakats commented Oct 1, 2026 •

Copy link
Copy Markdown
Member Author

Copilot review overview

🔵 Needs a closer look

The Linux workflow still installs and caches rustls under ~/rustls instead of ~/pkg/rustls.

Yep, that's an outlier because it's a binary download.

It wouldn't need to be cached, but still useful as a pinning substitute.

vszakats added a commit that referenced this pull request Oct 2, 2026
…vements

To make it easy to switch runner machine and have the locally built
dependencies and their caches adapt automatically. Also to drop
maintaining local build logic for CPUs separately.

Also:

- add support for Fil-C on ARM64.

- add a revision number ('1') to the cache keys and IDs.
  To make it easy to bump when doing modifications like this, or want
  to make tests that need rebuilding them.
  Cherry-picked from #23214

- move local builds from `~/` to `~/pkg/`. To have them under a single
  root, and to eventually sync them with GHA/http3-linux.
  Except for Rustls and Fil-C, which are binary downloads.
  Cherry-picked from #23214

Closes #23233
@vszakats vszakats changed the title GHA/http3-linux, linux: move and sync up install directories of local builds GHA/http3-linux: sync up install directories of local builds Oct 2, 2026
@vszakats
vszakats force-pushed the gha-localbuild-dir branch from 15fff50 to 5225a31 Compare October 2, 2026 13:19
@github-actions github-actions Bot added the HTTP/3 h3 or quic related label Oct 2, 2026
@vszakats
vszakats force-pushed the gha-localbuild-dir branch from 5225a31 to 95326e6 Compare October 2, 2026 13:31
@vszakats
vszakats force-pushed the gha-localbuild-dir branch from 95326e6 to 37139e5 Compare October 2, 2026 13:57
@vszakats vszakats changed the title GHA/http3-linux: sync up install directories of local builds GHA/http3-linux: add CPU and revision number to cache keys Oct 2, 2026
@vszakats

vszakats commented Oct 2, 2026 •

Copy link
Copy Markdown
Member Author

OpenSSL 3 on arm64 H3 valgrind fails:

FAIL 2500: 'HTTP/3 GET:' HTTP, HTTP GET, HTTP/3, --resolve
FAIL 2501: 'HTTP/3 POST' HTTP, HTTP POST, HTTP/3, HTTPS
FAIL 2502: 'HTTP GET multiple over HTTP/3' HTTP, HTTP/3, multi, verbose logs, libtest
FAIL 2503: 'HTTP/3 header-api' HTTP, HTTP/3, HTTPS, --resolve, -w, %header

https://github.com/curl/curl/actions/runs/37025210843/job/110898198578?pr=23214#step:30:19556
https://github.com/curl/curl/actions/runs/37026860992/job/110906146775

Maybe a issue within OpenSSL?

edit: no, it's also in quiche + BoringSSL:
https://github.com/curl/curl/actions/runs/37025210843/job/110901192895?pr=23214

edit 2: and OpenSSL 4, but not LibreSSL/AWS-LC, or bare BoringSSL:
https://github.com/curl/curl/actions/runs/37026860992

OpenSSL 4

test 2500 [HTTP/3 GET:]
 valgrind ERROR ==87743== Conditional jump or move depends on uninitialised value(s)
==87743==    at 0x535B5E0: OPENSSL_strlcpy (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x539098F: OBJ_obj2txt (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x5355FF7: get_legacy_cipher_names (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53526DF: OPENSSL_LH_doall_arg_thunk (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x538E9DF: OBJ_NAME_do_all (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x5354D77: ossl_namemap_stored (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53236DF: evp_generic_fetch (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53B25D3: RAND_get0_primary (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53B286F: RAND_status (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x4A61313: rand_enough (openssl.c:808)
==87743==    by 0x4A6B7C3: ossl_random (openssl.c:5472)
==87743==    by 0x4A6C6B3: Curl_ssl_random (vtls.c:372)
==87743==    by 0x4A0D10F: randit (rand.c:118)
==87743==    by 0x4A0D1D3: Curl_rand_bytes (rand.c:156)
==87743==    by 0x49B51FB: Curl_hash_global_init (hash.c:70)
==87743==    by 0x499E3FB: global_init (easy.c:180)
==87743== 
==87743== Conditional jump or move depends on uninitialised value(s)
==87743==    at 0x535B5E0: OPENSSL_strlcpy (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x539098F: OBJ_obj2txt (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53560EF: get_legacy_md_names (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53526DF: OPENSSL_LH_doall_arg_thunk (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x538E9DF: OBJ_NAME_do_all (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x5354D8B: ossl_namemap_stored (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53236DF: evp_generic_fetch (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53B25D3: RAND_get0_primary (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x53B286F: RAND_status (in /home/runner/pkg/openssl/lib/libcrypto.so.4)
==87743==    by 0x4A61313: rand_enough (openssl.c:808)
==87743==    by 0x4A6B7C3: ossl_random (openssl.c:5472)
==87743==    by 0x4A6C6B3: Curl_ssl_random (vtls.c:372)
==87743==    by 0x4A0D10F: randit (rand.c:118)
==87743==    by 0x4A0D1D3: Curl_rand_bytes (rand.c:156)
==87743==    by 0x49B51FB: Curl_hash_global_init (hash.c:70)
==87743==    by 0x499E3FB: global_init (easy.c:180)
==87743== 

https://github.com/curl/curl/actions/runs/37026860992/job/110903869051#step:30:152

Maybe a random generartor seed/init/bootstrap issue?

The quiche issue seems to be quiche-specific, not BoringSSL, and a different one than the above:
https://github.com/curl/curl/actions/runs/37026860992/job/110906146167

test 2500 [HTTP/3 GET:]
 valgrind ERROR ==15736== Conditional jump or move depends on uninitialised value(s)
==15736==    at 0x523FAB0: <quiche::Connection>::do_stream_recv::<&mut [u8]> (lib.rs:5883)
==15736==    by 0x52ADAEB: stream_recv_buf<quiche::buffers::DefaultBufFactory, &mut [u8]> (lib.rs:5763)
==15736==    by 0x52ADAEB: stream_recv<quiche::buffers::DefaultBufFactory> (lib.rs:5716)
==15736==    by 0x52ADAEB: <quiche::h3::stream::Stream>::try_fill_buffer::<quiche::buffers::DefaultBufFactory> (stream.rs:592)
==15736==    by 0x52B1263: <quiche::h3::Connection>::process_readable_stream::<quiche::buffers::DefaultBufFactory> (mod.rs:2615)
==15736==    by 0x52B4183: <quiche::h3::Connection>::poll::<quiche::buffers::DefaultBufFactory> (mod.rs:2160)
==15736==    by 0x5280D27: quiche_h3_conn_poll (ffi.rs:133)
==15736==    by 0x4A6AB4F: cf_poll_events (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x4A6A0B3: cf_process_ingress (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x4A682F3: cf_quiche_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x497896B: Curl_cf_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x497B3AB: Curl_conn_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x4A2A733: Curl_xfer_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x4A054BF: xfer_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x4A051E3: Curl_req_send (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x49B319F: Curl_http (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x49EFEEF: multi_do (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)
==15736==    by 0x49EE9B7: multistate_do (in /home/runner/work/curl/curl/bld/lib/libcurl.so.4.8.0)

@vszakats vszakats changed the title GHA/http3-linux: add CPU and revision number to cache keys GHA/http3-linux: add CPU and revision number to cache keys, switch 2 jobs to arm64 Oct 2, 2026
@vszakats vszakats closed this in 6c72371 Oct 2, 2026
@vszakats
vszakats deleted the gha-localbuild-dir branch October 2, 2026 16:12
vszakats added a commit that referenced this pull request Oct 6, 2026
…gs with openssl and quiche

Allow building quiche and OpenSSL H3 jobs on arm64 runner machines, by
avoiding compiler optimization bugs that make runtests 2500, 2501, 2502,
2503 fail with valgrind.

- build quiche with `CARGO_PROFILE_RELEASE_OPT_LEVEL=0` on arm64.
  The next optimization level (1) reproduces the failure.
  Dropping the `--release` option from `cargo build` also works. Seen
  with quiche 0.30.0, rustc 1.98.1 (48a229cea 2026-09-01).
  FTR with optimizations disabled runtests runs at the same speed as
  optimized Intel, and pytests is still noticeably faster than Intel.

- build OpenSSL 3/4 with clang.
  With gcc-15/16 it also works to pass `--debug` to `./Configure`, or to
  set `CFLAGS=-O2`, when building on/for arm64.
  Seen with OpenSSL 4.0.3, 3.5.9, 3.6.5, ubuntu-26.04-arm runner,
  gcc 15.1.0, gcc 16-20260322-1ubuntu1.

- switch openssl-prev jobs to arm64. (Identical performance)

Refs:
https://doc.rust-lang.org/cargo/reference/config.html#command-line-overrides
https://doc.rust-lang.org/cargo/reference/config.html#profile
https://doc.rust-lang.org/cargo/reference/profiles.html#opt-level
https://doc.rust-lang.org/cargo/reference/profiles.html#release

Bug: #23214 (comment)
Follow-up to 6c72371 #23214
Fixes #23237

Closes #23275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build CI Continuous Integration HTTP/3 h3 or quic related tidy-up

Development

Successfully merging this pull request may close these issues.

2 participants