feat: faster parallel S3 file downloads - #5102

What this PR does / why we need it:

multiprocessing.Pool.map distributes tasks in batches (chunksize = ceil(N/(P*4))). When multiple large model shards end up in the same batch, they are downloaded sequentially by a single worker process while other processes sit idle.

For example, Qwen3-4B has 27 files with 4 worker processes. pool.map calculates chunksize = ceil(27/16) = 2, meaning tasks are dispatched in pairs. The files are listed in alphabetical order from S3:

task[0]:  preprocessor_config.json                                    (390 Bytes)
task[1]:  ~~.json                                                     (N bytes)
...
task[20]: model-00001-of-00003.safetensors                 (3,957 MB) - Same proccess
task[21]: model-00002-of-00003.safetensors                 (3,987 MB) - Same process
task[22]: model-00003-of-00003.safetensors                 (95 MB)
...

With chunksize=2, task[20] and task[21] can batched together and assigned to the same worker process. That single process downloads them sequentially (12s + 12s = 24s) while other processes finish their small files in under 1s and sit idle.

With imap_unordered(chunksize=1), each task is dispatched individually to the next available worker. The two large shards are picked up by different processes and downloaded in parallel (~12s total).

This pattern is common across LLM model repositories where large safetensor shards dominate the total download time alongside many small metadata/config files.

Which issue(s) this PR fixes:
Fixes #

Feature/Issue validation/testing:

• Existing unit tests pass (37 passed)

• E2E test on g6e.4xlarge with Qwen3-4B (~8GB, 3 shards): 20s → 11.5s

• E2E test on g6e.4xlarge with Qwen3-VL-8B-Instruct (~16GB, 4 shards): 33s → 16s

• Per-process analysis confirmed large shards are now assigned to different worker PIDs

• Logs (pool.map)

PID=9  model-00001-of-00003.safetensors 3775MB 11.96s 316MB/s
PID=9  model-00002-of-00003.safetensors 3803MB 11.87s 320MB/s  ← same PID
TOTAL: 24.08s

• Logs (imap_unordered)

PID=8  model-00001-of-00003.safetensors 3775MB 12.88s 293MB/s
PID=7  model-00002-of-00003.safetensors 3803MB 13.55s 281MB/s  ← different PID
TOTAL: 13.57s

Special notes for your reviewer:

• imap_unordered returns an iterator instead of a list, so it is wrapped with list() to maintain the same behavior as pool.map (wait for all tasks to complete before processing results).
• MockPool in test_s3_storage.py is updated with imap_unordered method.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Improve S3 model download speed using imap_unordered