Skip to content

[ROCm] torch 2.13 wheel: "Can't detect vectorized ISA for CPU" in torch.compile smoke test on non-ROCm image (regression vs 2.12.1) #189194

Closed
Closed
[ROCm] torch 2.13 wheel: "Can't detect vectorized ISA for CPU" in torch.compile smoke test on non-ROCm image (regression vs 2.12.1)#189194
Assignees
jeffdailyjithunnair-amd
Labels
bot-triagedThis is a label only to be used by the auto triage botmodule: binariesAnything related to official binaries that we release to usersmodule: inductormodule: regressionIt used to work, and now it doesn'tmodule: rocmAMD GPU support for Pytorchmodule: vectorizationRelated to SIMD vectorization, e.g., Vec256oncall: pt2triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone
2.14.0

Description

@atalman
atalman
opened on Jul 7, 2026

🐛 Describe the bug

The torch==2.13.0+rocm7.2 wheel fails the torch.compile smoke test with
RuntimeError: Can't detect vectorized ISA for CPU when it is imported/run on
a non-ROCm image (e.g. pytorch/almalinux-builder:cpu-main), where
torch.cuda.is_available() is False and the compile check exercises the
CPU / Inductor path.

This looks like a regression:

  • ✅ torch==2.12.1+rocm7.2 runs the same smoke test on the same
    environment and succeeds — it prints Picked CPU ISA VecAVX2 bit width 256.
  • ✅ torch==2.13.0+rocm7.2 succeeds when run on the ROCm builder image
    (pytorch/manylinux2_28-builder:rocm7.2).
  • ❌ torch==2.13.0+rocm7.2 fails on a non-ROCm image.

So the 2.13 ROCm wheel appears to depend on something present in the ROCm
builder image to detect the CPU vectorized ISA, whereas 2.12.1 did not.

Repro

# In a plain (non-ROCm) manylinux/almalinux CPU image:
pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/test/rocm7.2
cd .ci/pytorch
python smoke_test.py --package torchonly

Traceback

Testing smoke_test_compile for cpu and torch.float16
Testing smoke_test_compile for cpu and torch.float32
Testing smoke_test_compile for cpu and torch.float64
Traceback (most recent call last):
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 652, in <module>
    main()
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 638, in main
    smoke_test_cuda(
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 357, in smoke_test_cuda
    smoke_test_compile("cuda" if torch.cuda.is_available() else "cpu")
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 511, in smoke_test_compile
    raise RuntimeError("Can't detect vectorized ISA for CPU")
RuntimeError: Can't detect vectorized ISA for CPU

CI evidence

For comparison, 2.12.1+rocm7.2 on the same environment logs:

torch: 2.12.1+rocm7.2
...
Testing smoke_test_compile for cpu and torch.float32
Picked CPU ISA VecAVX2 bit width 256
Testing smoke_test_compile with mode 'max-autotune' for torch.float32
...
OK: Wheel tag(s) valid for installed torch: cp312-cp312-manylinux_2_28_x86_64

Why this matters

This blocks validating the ROCm wheels on the regular (non-ROCm) validation
image. The vec-ISA probe in torch/_inductor/cpu_vec_isa.py
(pick_vec_isa() / valid_vec_isa_list()) is returning an empty list on the
2.13 ROCm wheel in an environment where 2.12.1 detected AVX2, which suggests
either a change in the probe's toolchain requirements or a packaging change in
the 2.13 ROCm wheel.

Versions

  • torch: 2.13.0+rocm7.2 (test channel), Python 3.12, cp312-cp312-manylinux_2_28_x86_64
  • Regression vs: 2.12.1+rocm7.2 (works)

cc @ezyang @gchanan @kadeng @msaroufim @malfet @tinglvv @nWEIdia @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @jataylo @hongxiayang @naromero77amd @pragupta @jerrymannil @xinyazhang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @muchulee8 @amjames @aakhundov @coconutruben

Activity

  1. atalman
    added
    module: rocmAMD GPU support for Pytorch
    oncall: pt2
    module: inductor
    on Jul 7, 2026
  2. github-project-automation
    added this to PyTorch on ROCmon Jul 7, 2026
  3. atalman
    added this to the 2.13.1 milestone on Jul 7, 2026
  4. github-actions
    added
    module: binariesAnything related to official binaries that we release to users
    triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
    module: vectorizationRelated to SIMD vectorization, e.g., Vec256
    module: regressionIt used to work, and now it doesn't
    bot-triagedThis is a label only to be used by the auto triage bot
    on Jul 7, 2026
  5. atalman
    assigned
    jithunnair-amd
    and
    jeffdaily
    on Jul 7, 2026
  6. atalman
    added
    high priority
    on Jul 7, 2026
  7. pytorch-bot
    added
    triage review
    on Jul 7, 2026

  8. atalman commented on Jul 7, 2026

    @atalman
    atalman
    on Jul 7, 2026
    CollaboratorAuthor

    Same issue in PyTorch nightly :

    torch: 2.14.0.dev20260707+rocm7.2
ATen/Parallel:
	at::get_num_threads() : 36
	at::get_num_interop_threads() : 96
OpenMP 201511 (a.k.a. OpenMP 4.5)
	omp_get_max_threads() : 36
Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
	mkl_get_max_threads() : 36
Intel(R) MKL-DNN v3.12.0 (Git Hash 80afa71049cd69a3df32adcccb623b12cd7baa22)
std::thread::hardware_concurrency() : 192
Environment variables:
	OMP_NUM_THREADS : 36
	MKL_NUM_THREADS : 36
ATen parallel backend: OpenMP

Skip version check for channel None as stable version is None
Testing smoke_test_conv2d
Testing smoke_test_linalg on cpu
Numpy check skipped. Numpy is not installed.
Testing SDPA on cpu using type torch.float16
Testing smoke_test_compile for cpu and torch.float16
Testing smoke_test_compile for cpu and torch.float32
Testing smoke_test_compile for cpu and torch.float64
Traceback (most recent call last):
  File "/home/dev/pytorch/.ci/pytorch/smoke_test/smoke_test.py", line 652, in <module>
    main()
  File "/home/dev/pytorch/.ci/pytorch/smoke_test/smoke_test.py", line 638, in main
    smoke_test_cuda(
  File "/home/dev/pytorch/.ci/pytorch/smoke_test/smoke_test.py", line 357, in smoke_test_cuda
    smoke_test_compile("cuda" if torch.cuda.is_available() else "cpu")
  File "/home/dev/pytorch/.ci/pytorch/smoke_test/smoke_test.py", line 511, in smoke_test_compile
    raise RuntimeError("Can't detect vectorized ISA for CPU")
RuntimeError: Can't detect vectorized ISA for CPU

  9. 3 remaining items

  10. albanD
    removed
    triage review
    on Jul 13, 2026
  11. atalman
    removed
    high priority
    on Jul 15, 2026
  12. atalman
    modified the milestones: 2.13.1, 2.14.0 on Jul 24, 2026

  13. atalman commented on Aug 13, 2026

    @atalman
    atalman
    on Aug 13, 2026
    CollaboratorAuthor

    cc @jeffdaily this is still crashing in validations for release 2.14 : https://github.com/pytorch/test-infra/actions/runs/31656923339/job/94427126002#step:17:3940

  14. pytorchmergebot
    added 2 commits that reference this issue on Aug 13, 2026

    [ROCm] Rewrite bundled-lib NEEDED entries after all libs are copied in

    9e642bc

    [ROCm] Retry VecISA dlopen probe with import torch on cold load failure

    4eb9fe3
  15. pytorchmergebot
    closed this as completedin 74e49c1on Aug 14, 2026
  16. github-project-automation
    moved this from Todo to Done in PyTorch on ROCmon Aug 14, 2026
  17. pytorchmergebot
    added a commit that references this issue on Aug 14, 2026

    [ROCm] Rewrite bundled-lib NEEDED entries after all libs are copied in (

    4973620
  18. apullin
    added a commit that references this issue on Aug 14, 2026

    [ROCm] Retry VecISA dlopen probe with import torch on cold load failu…

    74e49c1
  19. atalman
    added 2 commits that reference this issue on Aug 17, 2026

    [ROCm] Rewrite bundled-lib NEEDED entries after all libs are copied in (

    f746b5f

    [ROCm] Retry VecISA dlopen probe with import torch on cold load failu…

    f387a6e
  20. orrangetabby17
    added 2 commits that reference this issue on Sep 2, 2026

    [ROCm] Retry VecISA dlopen probe with import torch on cold load failu…

    0d991f2

    [ROCm] Rewrite bundled-lib NEEDED entries after all libs are copied in (

    d6ef365
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bot-triagedThis is a label only to be used by the auto triage botmodule: binariesAnything related to official binaries that we release to usersmodule: inductormodule: regressionIt used to work, and now it doesn'tmodule: rocmAMD GPU support for Pytorchmodule: vectorizationRelated to SIMD vectorization, e.g., Vec256oncall: pt2triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions