PolarsNDJSONWriter and PolarsNDJSONReader declare that they accept file objects (file: IOBase | str | Path, source: str | Path | IOBase | bytes), but any BytesIO or StringIO raises AttributeError: '_io.BytesIO' object has no attribute 'decode' from get_file_metadata. On the writer this happens after write_ndjson has already written the rows, so a save that looks failed to the caller has in fact written the data.

Current behavior

• PolarsNDJSONWriter(file=io.BytesIO()).save_data(df) raises AttributeError, and the buffer then holds both rows.
• PolarsNDJSONReader(source=io.BytesIO(raw)).load_data(pl.DataFrame) raises the same AttributeError, although pl.read_ndjson reads that buffer fine.
• PolarsJSONWriter and PolarsJSONReader behave the same way.
• With raw bytes as source the load succeeds, but the whole file content ends up in metadata["file_metadata"]["path"].

Stack Traces

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "hamilton/plugins/polars_post_1_0_0_extensions.py", line 610, in save_data
    return utils.get_file_and_dataframe_metadata(self.file, data)
  File "hamilton/io/utils.py", line 127, in get_file_and_dataframe_metadata
    return {**get_file_metadata(path), **get_dataframe_metadata(df)}
  File "hamilton/io/utils.py", line 47, in get_file_metadata
    parsed = parse.urlparse(path)
  File "/usr/lib/python3.12/urllib/parse.py", line 394, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/lib/python3.12/urllib/parse.py", line 133, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/lib/python3.12/urllib/parse.py", line 117, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: '_io.BytesIO' object has no attribute 'decode'

Steps to replicate behavior

1. pip install "apache-hamilton==1.90.0" polars
2. Run this script:

import io
import platform
from importlib.metadata import version

import polars as pl

from hamilton.plugins.polars_post_1_0_0_extensions import (
    PolarsJSONReader,
    PolarsJSONWriter,
    PolarsNDJSONReader,
    PolarsNDJSONWriter,
)

print(f"python: {platform.python_version()}")
print(f"apache-hamilton: {version('apache-hamilton')}")
print(f"polars: {pl.__version__}")

df = pl.DataFrame({"a": [1, 2], "b": ["x", "y"]})

buf = io.BytesIO()
try:
    PolarsNDJSONWriter(file=buf).save_data(df)
    print("writer save_data(BytesIO): OK")
except Exception as e:
    print(f"writer save_data(BytesIO): {type(e).__name__}: {e}")
print(f"buffer content after save_data: {buf.getvalue()!r}")

raw = buf.getvalue()
try:
    out, _ = PolarsNDJSONReader(source=io.BytesIO(raw)).load_data(pl.DataFrame)
    print("reader load_data(BytesIO): OK", out.shape)
except Exception as e:
    print(f"reader load_data(BytesIO): {type(e).__name__}: {e}")

print("pl.read_ndjson(BytesIO):", pl.read_ndjson(io.BytesIO(raw)).shape)

_, meta = PolarsNDJSONReader(source=raw).load_data(pl.DataFrame)
print(f"reader load_data(bytes) metadata path: {meta['file_metadata']['path']!r}")

try:
    PolarsJSONWriter(file=io.BytesIO()).save_data(df)
    print("PolarsJSONWriter save_data(BytesIO): OK")
except Exception as e:
    print(f"PolarsJSONWriter save_data(BytesIO): {type(e).__name__}: {e}")
try:
    PolarsJSONReader(source=io.BytesIO(df.write_json().encode())).load_data(pl.DataFrame)
    print("PolarsJSONReader load_data(BytesIO): OK")
except Exception as e:
    print(f"PolarsJSONReader load_data(BytesIO): {type(e).__name__}: {e}")

print("expected: save_data returns metadata without raising, load_data returns a (2, 2) frame")

Output:

python: 3.12.3
apache-hamilton: 1.90.0
polars: 2.0.0

writer save_data(BytesIO): AttributeError: '_io.BytesIO' object has no attribute 'decode'
buffer content after save_data: b'{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'
reader load_data(BytesIO): AttributeError: '_io.BytesIO' object has no attribute 'decode'
pl.read_ndjson(BytesIO): (2, 2)
reader load_data(bytes) metadata path: b'{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'
PolarsJSONWriter save_data(BytesIO): AttributeError: '_io.BytesIO' object has no attribute 'decode'
PolarsJSONReader load_data(BytesIO): AttributeError: '_io.BytesIO' object has no attribute 'decode'

expected: save_data returns metadata without raising, load_data returns a (2, 2) frame

Library & System Information

• python 3.12.3
• apache-hamilton 1.90.0
• polars 2.0.0
• Linux (x86_64)

The failure is in Hamilton's metadata helper, not in polars. The same code is on main at fa786ea.

Expected behavior

A file object that the field types accept works: save_data writes to the buffer and returns metadata, and load_data returns the (2, 2) frame. If file objects are not meant to be supported, the classes should not declare IOBase and bytes, and the writer should not report failure after it has written.

Additional context

Cause, with lines at main fa786ea, in hamilton/plugins/polars_post_1_0_0_extensions.py: the reader's source: str | Path | IOBase | bytes (line 566) is passed to utils.get_file_metadata(self.source) at line 584. The writer's file: IOBase | str | Path (line 599) is written with data.write_ndjson(self.file) at line 609 and then passed to utils.get_file_and_dataframe_metadata(self.file, data) at line 610. get_file_metadata (hamilton/io/utils.py line 69) is typed for paths, converts only Path to str, and calls parse.urlparse(path) at line 83 (line 47 in the 1.90.0 traceback above), which fails on any file object. PolarsJSONReader (line 526) and PolarsJSONWriter (line 552) have the same pattern. The existing NDJSON tests only round-trip a file path, so this path is not covered.

Proposed fix: compute file metadata only when source / file is a str or Path, and return dataframe-only metadata for file objects and raw bytes. The alternative is to remove IOBase and bytes from the declared field types. Either way, add a test that saves to and loads from a BytesIO.

I'm happy to take this one and will open a PR with the fix and the test.