Fix http_get_file deleting correctly downloaded files (gzip, chunked, resumed 206) - #1802

Problem

modelscope/hub/file_download.py::http_get_file() finishes with

if total != downloaded_length:
    os.remove(temp_file.name)
    raise FileDownloadError('... download incomplete ...')

where total is the response's Content-Length. That header does not describe the size of the file written to disk in three common cases, and in each of them a complete, byte-exact download is deleted and reported as incomplete:

requests sends Accept-Encoding: gzip, deflate by default, so any origin with compression enabled hits the first case with no special configuration. The third case needs a download over one chunk (1 MB) that gets interrupted once — i.e. large files on a flaky network, which is exactly when resume matters. http_get_file() is used for arbitrary URLs by preprocessors/video.py, models/cv/action_detection/action_detection_onnx.py, models/multi_modal/prost/models/prost_model.py (uncaught → pipeline crashes) and models/multi_modal/mmr/models/clip_for_mm_video_embedding.py (caught → returns an all-zero embedding).

Fix

Only compare against an expected on-disk size when one can be established: identity encoding with a Content-Length → that length; 206 → the complete length parsed from Content-Range: bytes s-e/complete; chunked or non-identity encoding → skip the comparison. Skipping does not lose the truncation check: a response shorter than its declared Content-Length already raises IncompleteRead inside iter_content (urllib3 enforces it), which is what triggers the retry loop today.

The 206 handling deliberately uses Content-Range's complete length rather than downloaded + Content-Length: a server that answers 206 but restarts from byte 0 would otherwise produce a corrupt file that passes the check. That case is covered by a negative test.

Tests

tests/hub/test_http_get_file.py — five cases against a local http.server, no network or credentials, ~3 s:

• gzip-encoded response → file content equals the payload
• chunked response without Content-Length → file content equals the payload
• interrupted download resumed with 206 → file content equals the payload, more than one request made
• 206 restarting from the wrong offset → still rejected (FileDownloadError), file removed
• truncated body → still rejected, file removed

On master the first three error out (the file is deleted), the two negative cases pass; with this change all five pass. Additionally, 14 well-formed identity responses (0, 1, 4095, 1 MB−1, 1 MB, 1 MB+1, 3 MB+7 bytes, with and without an explicit Content-Encoding: identity) produce byte-identical results and hashes before and after the change.

flake8 / yapf clean; isort applied to the new test file (file_download.py's existing import block already fails the repo's isort settings on master, left untouched).

Note: I could not run the full test suite locally because the installed modelscope_hub wheel is older than the repo's pin; tests/ has no other coverage of http_get_file, and the change stays inside that function plus one private helper.

Written with Claude Code (AI-assisted) and submitted under the account owner's authorization.