Numeric fill_value when loading a DataFrame with a MultiIndex and string columns fail with Pandas >= 3 #1166

Activity

1. gdementen

   added this to the 0.35.1 milestone on Mar 4, 2026
2. gdementen

   added

   bug

   task

   priority: very high

   on Mar 4, 2026
3. gdementen commented on Mar 6, 2026

   

   gdementen

   on Mar 6, 2026

   ContributorAuthor

   More actions

   It turns out it is because of some data (read from Excel but that's irrelevant) has a multi-index with some missing combination of values (when expanded to an array) and a fill_value not compatible with the dtype of some columns (in the model a fill_value of zero while the column is string). In the user code, the string column is dropped just after reading the array.

   Here is an MRE:

   >>> df = pd.DataFrame({
   ...     'idx1': [0, 0, 1],
   ...     'idx2': ['a', 'b', 'a'],
   ...     'col1': ['a', 'a', 'a'],
   ...     'col2': [1, 1, 1]
   ... }).set_index(['idx1', 'idx2'])
   >>> res = from_frame(df, fill_value=0)
   TypeError: Invalid value '0' for dtype 'str'. Value should be a string or missing value, got 'int' instead.

   It previously worked because Pandas an object dtype for string columns and thus the 0 fill_value worked. This is an ugly issue and I don't really know what's the best way to solve this.
4. gdementen

   removed

   task

   on Mar 6, 2026
5. gdementen

   changed the title [-]demo model crashes on Pandas >= 3[/-] [+]Numeric fill_value when loading a DataFrame with a MultiIndex and string columns fail with Pandas >= 3[/+] on Mar 6, 2026
6. gdementen commented on Mar 6, 2026

   

   gdementen

   on Mar 6, 2026

   ContributorAuthor

   More actions

   The best way to solve this would probably be to implement a "usecols" argument in some (all?) functions and methods involved (read_excel, Range.load(), from_lists and from_frame) but that wouldn't solve the immediate problem.
7. gdementen

   self-assigned this

   on Mar 10, 2026
8. gdementen

   added

   work in progress

   on Mar 10, 2026
9. gdementen

   added 2 commits that reference this issue on Mar 27, 2026

   FIX: MultiIndex+str column with Pandas 3.0 (fixes larray-project#1166)

   This issue will close once commit 8b9f5e1 is merged into the 'master' branch.

   8b9f5e1

   FIX: MultiIndex+str column with Pandas 3.0 (fixes larray-project#1166)

   This issue will close once commit 0189dab is merged into the 'master' branch.

   0189dab
10. gdementen

    added a commit that references this issue on Apr 3, 2026

    FIX: MultiIndex+str column with Pandas 3.0 (fixes larray-project#1166)

    This issue will close once commit 671736b is merged into the 'master' branch.

    671736b
11. gdementen

    closed this as completedin cfbbe06on Apr 3, 2026
12. gdementen

    added a commit that references this issue on Apr 14, 2026

    FIX: MultiIndex+str column with Pandas 3.0 (fixes larray-project#1166)

    This issue will close once commit cfbbe06 is merged into the 'master' branch.

    cfbbe06
13. gdementen

    removed

    work in progress

    on Apr 14, 2026