Skip to content

Fix array shuffle index overflow for large groups - #12590

Open
AyoobMH wants to merge 1 commit into
dask:mainfrom
AyoobMH:fix/shuffle-large-group-overflow
Open

AyoobMH wants to merge 1 commit into
dask:mainfrom
AyoobMH:fix/shuffle-large-group-overflow

Conversation

@AyoobMH

@AyoobMH AyoobMH commented Sep 8, 2026 •

Copy link
Copy Markdown

AI-assisted contribution, reviewed by @AyoobMH, who has confirmed understanding of the changes.

Array shuffle can wrap its sorting indices when a group is larger than both the input chunks and the target chunk-size limit. For example, 20-element input chunks with 257-element groups can silently return incorrect values; larger groups can instead raise the misleading unknown-chunk-size error. Include the actual output-group sizes when choosing the index dtype.

Addresses pydata/xarray#11212, reported by @joshua-gould, and extends the dtype bound introduced in #11615. The regression tests cover the uint8 boundary, uint16 overflow, both array axes, and repeated indices.

  • Tests added / passed: shuffle and slicing suites — 146 passed, 2 skipped, 3 xfailed. Before the fix, seven new cases failed and the two boundary controls passed.
  • Xarray reproduction and twelve diagnostic boundary/integration checks pass with the local Dask fix.
  • Passes pixi run lint (not run; pinned Ruff and Black checks and git diff --check passed for the changed files).

@AyoobMH AyoobMH changed the title [unsupervised AI] Fix array shuffle index overflow for large groups Fix array shuffle index overflow for large groups Sep 8, 2026
@AyoobMH
AyoobMH marked this pull request as ready for review September 8, 2026 19:44
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

     25 files  ±  0       25 suites  ±0   7h 35m 32s ⏱️ + 25m 48s
 19 923 tests +  9   18 490 ✅ +  9   1 433 💤 ± 0  0 ❌ ±0 
379 551 runs  +198  332 262 ✅ +181  47 289 💤 +17  0 ❌ ±0 

Results for commit 6612aa0. ± Comparison against base commit 9dc535d.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant