The HTTP Range header lets you pass bytes=-524288 to get the last 512K bytes of a remote resource.
Instead of first fetching the bytelength with a HEAD request, then fetch the metadata by computing the start-end bytes range, we can directly pass bytes=-524288. If it's supported by the server, it saves one request, since the response should include the bytelength in the content-range response header.
Eg:
curl https://github.com/hyparam/hyparquet/raw/6f5ac750cd24c518cf02d73c36847b5f6045cafa/test/files/strings.parquet -i -H "Range: bytes=-524288"
...
content-range: bytes 222481287-223005574/223005575
I'm not sure it's easy to add to the code, though, because the file option we pass to the functions does not know if it's for a remote URL or a local file.
Also, I'm not sure all the servers support this syntax, so if it's not wide spread, it might increase the latency instead.
The HTTP Range header lets you pass
bytes=-524288to get the last 512K bytes of a remote resource.Instead of first fetching the bytelength with a HEAD request, then fetch the metadata by computing the
start-endbytes range, we can directly passbytes=-524288. If it's supported by the server, it saves one request, since the response should include the bytelength in thecontent-rangeresponse header.Eg:
curl https://github.com/hyparam/hyparquet/raw/6f5ac750cd24c518cf02d73c36847b5f6045cafa/test/files/strings.parquet -i -H "Range: bytes=-524288" ... content-range: bytes 222481287-223005574/223005575I'm not sure it's easy to add to the code, though, because the
fileoption we pass to the functions does not know if it's for a remote URL or a local file.Also, I'm not sure all the servers support this syntax, so if it's not wide spread, it might increase the latency instead.