provide converters for datetime64 types

See https://travis-ci.org/matplotlib/matplotlib/jobs/294376461#L2096 for more details

Possibly due to https://pandas.pydata.org/pandas-docs/version/0.21/whatsnew.html#dtype-conversions

Hmm, actually seems to be because the units aren't being converted by the call to self.convert_xunits

The issue is this line:

Line 5248 in b0a0a8e

X, Y, C = [np.asanyarray(a) for a in args]

Running a 'pandas.core.indexes.datetimes.DatetimeIndex' object through np.asanyarray now returns a 'numpy.ndarray', so the unit conversion isn't done properly later. I don't have time to throw together a fix, but hopefully this helps!

Thats a bother. Particularly as we don't like numpy.datetime64 which is the dtype it gets converted to.

A fix is to check for datetime64 arrays after the call above, and convert (back) to pandas after....

Another fix is for the units registry to recognize datetime64 arrays and convert there....

Another fix is for the units registry to recognize datetime64 arrays and convert there....

Pandas does it in their registry code: link

def register():
    units.registry[lib.Timestamp] = DatetimeConverter()
    units.registry[Period] = PeriodConverter()
    units.registry[pydt.datetime] = DatetimeConverter()
    units.registry[pydt.date] = DatetimeConverter()
    units.registry[pydt.time] = TimeConverter()
    units.registry[np.datetime64] = DatetimeConverter()

so I'm wondering if this is an upstream fix that needs to happen on their end...

@story645

@story645 Hmmm, very confused. Do we call their converter? It looks like they rewrote matplotlib's for their plotting?

No, I don't think so, though we refer to their converter as a good way to get from np.datetime64 to datetime. https://matplotlib.org/faq/howto_faq.html#plot-numpy-datetime64-values

I wonder why we don't just use this ourselves?

The point is that they used to register their converter when pandas was imported; now they do it only when they need it for their plotting. This makes sense in that having our code changed by the simple act of importing pandas is intrusive.
Their converter is their own, not a rewrite of ours. We have never added datetime64 support of our own.

The don't only import it on plotting because they import their convertor in their timeseries code

from pandas.plotting._converter import (register, time2num,
                                        TimeConverter, TimeFormatter,
                                        PeriodConverter, get_datevalue,
                                        DatetimeConverter,
                                        PandasAutoDateFormatter,
                                        PandasAutoDateLocator,
                                        MilliSecondLocator, get_finder,
                                        TimeSeries_DateLocator,
                                        TimeSeries_DateFormatter)

so I think we're still hitting it. Otherwise I think plotting would be totally broken for pandas timeseries objects if not done through their plotting api.

Eta: matplotlib's dates.py only supports datetime objects

units.registry[datetime.date] = DateConverter()
units.registry[datetime.datetime] = DateConverter()

@efiring

OK, I'm slowly understanding...

I run

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

time = pd.date_range('2000-01-01', periods=20)
depth = np.arange(20)
data = np.random.rand(20, 20)

fig, ax = plt.subplots()
ax.pcolormesh(time, depth, data)

For pandas-0.20 things run fine, and the units registry has keys:

dict_keys([<class 'datetime.date'>, <class 'datetime.datetime'>, 
<class 'pandas._libs.tslib.Timestamp'>, <class 'pandas._libs.period.Period'>, 
<class 'datetime.time'>, <class 'numpy.datetime64'>, <class 'str'>, <class 'numpy.str_'>, 
<class 'bytes'>, <class 'numpy.bytes_'>])

For pandas-0.21, we get the error, and the units registry only has keys:

dict_keys([<class 'str'>, <class 'numpy.str_'>, <class 'bytes'>, <class 'numpy.bytes_'>, 
<class 'datetime.date'>, <class 'datetime.datetime'>])

If I include

import pandas.plotting._converter as pandacnv
pandacnv.register()

then I get back the pandas stuff and the example runs fine.

dict_keys([<class 'str'>, <class 'numpy.str_'>, <class 'bytes'>, <class 'numpy.bytes_'>,
 <class 'datetime.date'>, <class 'datetime.datetime'>, <class 'pandas._libs.tslib.Timestamp'>, 
<class 'pandas._libs.period.Period'>, <class 'datetime.time'>, <class 'numpy.datetime64'>])

So, as @efiring said, what happened here was that pandas 0.21 moved the import of their registers out of the base import, and into their plotting area. More to the point, they now explicitly require the call of register().

FWIW, our pandas-0.21 plotting is indeed broken for other examples:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

time = pd.date_range('2000-01-01', periods=20)
depth = np.arange(20)
data = np.random.rand(20, 20)

fig, ax = plt.subplots()
ax.plot(time, data[3,:])
plt.show()

... doesn't outright fail but gets:

Whereas it is decoded correctly in pandas-0.20.

Yeah, pandas.plotting.__init__ used to call https://github.com/pandas-dev/pandas/blob/3a7f956c30528736beaae5784f509a76d892e229/pandas/plotting/__init__.py#L7

    from pandas.plotting import _converter
    _converter.register()  # needs to override so set_xlim works with str/number

But that was removed in 0.21 (pandas-dev/pandas#17710)

Not at all sure what to do here.

We could call _converter.register() from dates.py and load up all their calls. Pandas older than 0.20 will not work.
We could implement our own handling of datetime64 support.
We could tell panda's users to run _converter.register() themselves (and update our examples/tests).
I guess we could ask Pandas to go back to making the import pandas register their units.

Option 1 May break examples and code that didn't load pandas.
Option 2 May break some pandas examples
Option 3 Is inelegant for pandas users who are bypassing panda's plotting functions.

@TomAugspurger

See also pandas-dev/pandas#18153 Looks like the advice there (@TomAugspurger) is to use the extra verbiage.

I agree it's in our court, but I also think that doing ax.plot(data['Displacement']) where data is a DataFrame has a datetime index should be possible without faffing around with registers, since lots of scientific data is time series.

I think this should be re-framed as datetime64 support in mpl, not pandas support. Pandas uses one very restricted flavor of datetime64.

Changed the title, moved to 2.2 and updated the labels.

@tacaswell

There actually already was an issue for "datetime64 support in mpl": #1097, and which was closed recently deferring to pandas ... (but of course, situation has changed in the meantime)

@tacaswell I looked into doing that a while ago and iirc it very quickly ended up in the pandas c-extensions and internal details of the time objects in pandas....

@TomAugspurger I also once looked at it, and think agree with this that it is not that easy to port the full pandas functionality (although it is a long time ago).
But I think adding a basic Converter that just understands numpy datetime64 and converts this to matplotlib's internal float date numbers should be rather easy. The bulk of the code in the pandas implementation is the 'smarter' locator/formatter for fixed frequency data. But that might not be necessary for matplotlib (or at least not for a first basic support)

FYI, given the many feedback we got about this, we are considering on the pandas side to temporarily undo this change in the upcoming bug fix release to have a smoother deprecation period of the automatic registering of our converters: pandas-dev/pandas#18301

Closed by #9779

Navigation Menu

Repository navigation

provide converters for datetime64 types #9610

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

provide converters for datetime64 types #9610

Description

Activity

dstansby commented on Oct 29, 2017

dstansby commented on Oct 29, 2017

dstansby commented on Oct 29, 2017

jklymak commented on Nov 9, 2017

story645 commented on Nov 9, 2017

jklymak commented on Nov 9, 2017

jklymak commented on Nov 9, 2017

efiring commented on Nov 9, 2017

story645 commented on Nov 9, 2017

jklymak commented on Nov 9, 2017

jklymak commented on Nov 9, 2017

jklymak commented on Nov 9, 2017

18 remaining items

dstansby commented on Nov 13, 2017

efiring commented on Nov 13, 2017

tacaswell commented on Nov 13, 2017

jorisvandenbossche commented on Nov 13, 2017

jorisvandenbossche commented on Nov 15, 2017

tacaswell commented on Nov 20, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions