Skip to content

Commit b7735f6

Browse files
Backport PR #66117 on branch 3.0.x (BUG: interpolate leaving NAs unfilled for pyarrow dtypes) (#68924)
Co-authored-by: Apoorv Darshan <ad13dtu@gmail.com>
1 parent bca0b77 commit b7735f6

3 files changed

Lines changed: 25 additions & 13 deletions

File tree

‎doc/source/whatsnew/v3.0.6.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Fixed regressions
1919
- Bug in :meth:`Timestamp.tz_convert` and :meth:`Timestamp.tz_localize` and their :class:`Index` and :class:`Series` counterparts returning a UTC offset a few minutes off for dates before the specified timezone adopted a standard time (typically early 20th century); this only affected older systems whose ``tzdata`` was compiled with a toolchain from between 2013 and 2018, such as Amazon Linux 2 and RHEL 8 and its derivatives (:issue:`67066`)
2020
- Fixed bug in :func:`read_csv` with the ``c`` engine and ``low_memory=True`` where a mixed-dtype column selected by ``usecols`` raised ``IndexError`` instead of emitting a :class:`DtypeWarning` unless it was the first column in the file, and where that warning named the wrong column when the file had an index column (:issue:`67375`)
2121
- Fixed bug in :func:`read_csv` with the ``c`` engine and ``low_memory=True`` where the number reported next to each column name in the mixed-type :class:`DtypeWarning` was a running count of the warned columns rather than the column's position in the file (:issue:`67375`)
22+
- Regression in :meth:`Series.interpolate` with ``method="linear"`` and a pyarrow-backed dtype leaving consecutive and trailing missing values unfilled, and truncating interpolated values for integer dtypes; the result now matches the equivalent masked (e.g. ``Int64``) dtype, including the upcast to ``float64[pyarrow]`` (:issue:`65345`)
2223

2324
.. ---------------------------------------------------------------------------
2425
.. _whatsnew_306.bug_fixes:

‎pandas/core/arrays/arrow/array.py‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2507,18 +2507,11 @@ def interpolate(
25072507
if not self.dtype._is_numeric:
25082508
raise TypeError(f"Cannot interpolate with {self.dtype} dtype")
25092509

2510-
if (
2511-
method == "linear"
2512-
and limit_area is None
2513-
and limit is None
2514-
and limit_direction == "forward"
2515-
):
2516-
values = self._pa_array.combine_chunks()
2517-
na_value = pa.array([None], type=values.type)
2518-
y_diff_2 = pc.fill_null_backward(pc.pairwise_diff_checked(values, period=2))
2519-
prev_values = pa.concat_arrays([na_value, values[:-2], na_value])
2520-
interps = pc.add_checked(prev_values, pc.divide_checked(y_diff_2, 2))
2521-
return self._from_pyarrow_array(pc.coalesce(self._pa_array, interps))
2510+
# GH#65345: a pyarrow-native fast path for
2511+
# method="linear"/limit_direction="forward" was removed here because
2512+
# it only handled isolated NAs (leaving consecutive and trailing NAs
2513+
# unfilled), truncated interpolated values for integer dtypes, and
2514+
# did not upcast to float64 like the general path below.
25222515

25232516
mask = self.isna()
25242517
if self.dtype.kind == "f":

‎pandas/tests/extension/test_arrow.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3705,9 +3705,27 @@ def test_interpolate_not_numeric(data):
37053705

37063706
@pytest.mark.parametrize("dtype", ["int64[pyarrow]", "float64[pyarrow]"])
37073707
def test_interpolate_linear(dtype):
3708+
# GH#65345 results should match the masked (e.g. Int64) dtypes:
3709+
# upcast to float, and fill the trailing NA going forward
37083710
ser = pd.Series([None, 1, 2, None, 4, None], dtype=dtype)
37093711
result = ser.interpolate()
3710-
expected = pd.Series([None, 1, 2, 3, 4, None], dtype=dtype)
3712+
expected = pd.Series([None, 1.0, 2.0, 3.0, 4.0, 4.0], dtype="float64[pyarrow]")
3713+
tm.assert_series_equal(result, expected)
3714+
3715+
3716+
def test_interpolate_linear_consecutive_na():
3717+
# GH#65345 consecutive interior NAs were left unfilled
3718+
ser = pd.Series([1, 2, 3, None, None, 6, 7], dtype="int64[pyarrow]")
3719+
result = ser.interpolate(method="linear", limit_direction="forward")
3720+
expected = pd.Series([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0], dtype="float64[pyarrow]")
3721+
tm.assert_series_equal(result, expected)
3722+
3723+
3724+
def test_interpolate_linear_int_fractional():
3725+
# GH#65345 result should not truncate the interpolated value (1 instead of 1.5)
3726+
ser = pd.Series([1, None, 2], dtype="int64[pyarrow]")
3727+
result = ser.interpolate(method="linear")
3728+
expected = pd.Series([1.0, 1.5, 2.0], dtype="float64[pyarrow]")
37113729
tm.assert_series_equal(result, expected)
37123730

37133731

0 commit comments

Comments
 (0)