Describe the issue:
During merging of dask dataframes you can get a warning which states you have a type mismatch showing the same type:
./venv/lib/python3.11/site-packages/dask/dataframe/multi.py:169: UserWarning: Merging dataframes with merge column data type mismatches:
+---------------------------------+------------+-------------+
| Merge columns | left dtype | right dtype |
+---------------------------------+------------+-------------+
| ('Bank Name Last Part', 'City') | string | string |
+---------------------------------+------------+-------------+
Cast dtypes explicitly to avoid unexpected results.
It advises to cast columns explicitly. But the left column is the result of astype('string') and the right column is got after read_csv('file.csv', dtype='string').
If we check ddf.dtypes we notice that types aren't completely the same. They are string[pyarrow] and string[python].
Are they compatible? I suppose if they are this warning shouldn't be printed and if they are not the warning should print full types instead of just string.
Minimal Complete Verifiable Example:
import dask.dataframe as dd
import pandas as pd
for lib in [pd, dd]:
print(f'= lib: {lib.__name__} =')
cities = lib.read_csv('banklist.csv', dtype='string')[['City', 'ST']].drop_duplicates()
banks = lib.read_csv('banklist.csv', dtype='string')[['Bank Name', 'Acquiring Institution']]
banks['Bank Name Last Part'] = banks['Bank Name'].str.split(' ').str[0].astype('string')
print('= cities =', cities.dtypes, sep='\n')
print('= banks =', banks.dtypes, sep='\n')
banks_with_geo = banks.merge(cities, left_on='Bank Name Last Part', right_on='City', how='left')
print('= banks_with_geo =', banks_with_geo.dtypes, sep='\n')
print()Anything else we need to know?:
There was a similar bug with this function already #5437.
Example data: banklist.csv.
MCVE output:
= lib: pandas =
= cities =
City string[python]
ST string[python]
dtype: object
= banks =
Bank Name string[python]
Acquiring Institution string[python]
Bank Name Last Part string[python]
dtype: object
= banks_with_geo =
Bank Name string[python]
Acquiring Institution string[python]
Bank Name Last Part string[python]
City string[python]
ST string[python]
dtype: object
= lib: dask.dataframe =
= cities =
City string[pyarrow]
ST string[pyarrow]
dtype: object
= banks =
Bank Name string[pyarrow]
Acquiring Institution string[pyarrow]
Bank Name Last Part string[python]
dtype: object
./venv/lib/python3.11/site-packages/dask/dataframe/multi.py:169: UserWarning: Merging dataframes with merge column data type mismatches:
+---------------------------------+------------+-------------+
| Merge columns | left dtype | right dtype |
+---------------------------------+------------+-------------+
| ('Bank Name Last Part', 'City') | string | string |
+---------------------------------+------------+-------------+
Cast dtypes explicitly to avoid unexpected results.
warnings.warn(
= banks_with_geo =
Bank Name string[pyarrow]
Acquiring Institution string[pyarrow]
Bank Name Last Part object
City string[pyarrow]
ST string[pyarrow]
dtype: object
Environment:
- Dask version: 2025.4.1
- Python version: 3.11.5
- Operating System: Linux
- Install method (conda, pip, source): pip
Describe the issue:
During merging of dask dataframes you can get a warning which states you have a type mismatch showing the same type:
It advises to cast columns explicitly. But the left column is the result of
astype('string')and the right column is got afterread_csv('file.csv', dtype='string').If we check
ddf.dtypeswe notice that types aren't completely the same. They arestring[pyarrow]andstring[python].Are they compatible? I suppose if they are this warning shouldn't be printed and if they are not the warning should print full types instead of just
string.Minimal Complete Verifiable Example:
Anything else we need to know?:
There was a similar bug with this function already #5437.
Example data: banklist.csv.
MCVE output:
Environment: