Skip to content

Commit 6377e4c

Browse files
andersbogsnesandersbogsnes
committed
Merge branch 'release/v0.12.0'
2 parents 4775e7e + f8aad49 commit 6377e4c

76 files changed

Lines changed: 3556 additions & 1756 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ jobs:
99
runs-on: ubuntu-latest
1010
steps:
1111
- uses: actions/checkout@v2
12-
- uses: actions/setup-python@v2.1.1
12+
- uses: actions/setup-python@v2.1.2
1313
with:
1414
python-version: 3.8
1515
- uses: dschep/install-poetry-action@v1.3

‎.github/workflows/test.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: Integration
2-
on: push
2+
on: pull_request
33
env:
44
TOX_VERSION: 3.17.1
55
POETRY_VERSION: 1.0.9
@@ -13,7 +13,7 @@ jobs:
1313
name: Test Python ${{ matrix.python-version }} on ${{ matrix.os }}
1414
steps:
1515
- uses: actions/checkout@v2
16-
- uses: actions/setup-python@v2.1.1
16+
- uses: actions/setup-python@v2.1.2
1717
with:
1818
python-version: ${{ matrix.python-version }}
1919
- uses: BSFishy/pip-action@v1
@@ -27,7 +27,7 @@ jobs:
2727
runs-on: ubuntu-latest
2828
steps:
2929
- uses: actions/checkout@v2
30-
- uses: actions/setup-python@v2.1.1
30+
- uses: actions/setup-python@v2.1.2
3131
with:
3232
python-version: 3.8
3333
- uses: pre-commit/action@v2.0.0
@@ -42,7 +42,7 @@ jobs:
4242
name: Documentation python ${{ matrix.python-version }}
4343
steps:
4444
- uses: actions/checkout@v2
45-
- uses: actions/setup-python@v2.1.1
45+
- uses: actions/setup-python@v2.1.2
4646
with:
4747
python-version: ${{ matrix.python-version }}
4848
- uses: dschep/install-poetry-action@v1.3
@@ -56,7 +56,7 @@ jobs:
5656
runs-on: ubuntu-latest
5757
steps:
5858
- uses: actions/checkout@v2
59-
- uses: actions/setup-python@v2.1.1
59+
- uses: actions/setup-python@v2.1.2
6060
with:
6161
python-version: 3.8
6262
- uses: dschep/install-poetry-action@v1.3
@@ -65,4 +65,4 @@ jobs:
6565
- run: pip install tox==$TOX_VERSION
6666
- run: tox -e py,optional-artifactory,coverage-report
6767
- name: Upload coverage to Codecov
68-
uses: codecov/codecov-action@v1.0.12
68+
uses: codecov/codecov-action@v1.0.13

‎CHANGELOG.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,20 @@
1+
# v0.12.0
2+
- Permutation importance and Feature importance are now two different plotting methods.
3+
- `Model.test_estimators` now takes a `feature_pipeline` argument
4+
- Fixed a bug where `FillNA` did not create a `_is_na` column if the column didn't have a missing value
5+
- Implemented Bayesian Search for hyperparameter optimization
6+
- Added a `read_file` convenience method to `FileDataset` to read
7+
- Fixed a bug where `copy_to` failed between two instances of Sqlite based SQLDatasets
8+
- Fixed a bug where `ClassificationVisualize.confusion_matrix` would fail on multi-class problems due to wrong defaults
9+
- Added __repr__ to demodataset
10+
- Lift curve now can plot multi-class
11+
- Precision-Recall curve can now plot multi-class
12+
- ROC AUC curve can now plot multi-class
13+
- Fixed Binner to have a default value
14+
- Fixed FuncTransform to have a default value
15+
- `load_estimator` now uses default storage if nothing is passed
16+
- `Model.bayessearc` is now `Model.bayesiansearch`
17+
118
# v0.11.0
219
- Added `load_demo_dataset` function
320
- If the dataset has no train set `score_estimator` will now run `create_train_test` with default configurations
@@ -9,6 +26,7 @@
926
before visualization
1027
- New config implementation. If you need to reset the configuration, you should use `Model.config.reset_config()`
1128

29+
1230
# v0.10.3
1331
- Fixed typehints in Dataset
1432
- Dataset.create_train_test now takes a boolean `stratify` parameter.

‎RELEASE_CHECKLIST.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Release Checklist
2-
- [ ] Update version number in setup.cfg
2+
- [ ] Update version number in setup.py
33
- [ ] Update CHANGELOG
4-
- [ ] Run `python setup.py build`
5-
- [ ] Update conda feedstock
4+
- [ ] Run `python setup.py bdist_wheel sdist`
65
- [ ] run `twine upload`
6+
- [ ] Update conda feedstock

‎dev-requirements.txt‎

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,44 +5,46 @@
55
# pip-compile dev-requirements.in
66
#
77
appdirs==1.4.4 # via virtualenv
8-
attrs==19.3.0 # via -r requirements.txt, pytest
9-
certifi==2020.6.20 # via requests
8+
attrs==20.1.0 # via -r requirements.txt, pytest
9+
certifi==2020.6.20 # via matplotlib, requests
1010
cfgv==3.1.0 # via pre-commit
1111
chardet==3.0.4 # via requests
12-
coverage==5.2 # via pytest-cov
12+
coverage==5.2.1 # via pytest-cov
1313
cycler==0.10.0 # via -r requirements.txt, matplotlib
1414
distlib==0.3.1 # via virtualenv
1515
dohq-artifactory==0.7.382 # via -r dev-requirements.in
1616
filelock==3.0.12 # via tox, virtualenv
1717
identify==1.4.25 # via pre-commit
1818
idna==2.10 # via requests
19-
joblib==0.16.0 # via -r requirements.txt, scikit-learn
19+
iniconfig==1.0.0 # via pytest
20+
joblib==0.16.0 # via -r requirements.txt, scikit-learn, scikit-optimize
2021
kiwisolver==1.2.0 # via -r requirements.txt, matplotlib
21-
matplotlib==3.3.0 # via -r requirements.txt
22+
matplotlib==3.3.1 # via -r requirements.txt
2223
more-itertools==8.4.0 # via pytest
2324
nodeenv==1.4.0 # via pre-commit
24-
numpy==1.19.1 # via -r requirements.txt, matplotlib, pandas, scikit-learn, scipy
25+
numpy==1.19.1 # via -r requirements.txt, matplotlib, pandas, scikit-learn, scikit-optimize, scipy
2526
packaging==20.4 # via pytest, tox
26-
pandas==1.0.5 # via -r requirements.txt
27+
pandas==1.1.1 # via -r requirements.txt
2728
pillow==7.2.0 # via -r requirements.txt, matplotlib
2829
pluggy==0.13.1 # via pytest, tox
29-
pre-commit==2.6.0 # via -r dev-requirements.in
30+
pre-commit==2.7.1 # via -r dev-requirements.in
3031
py==1.9.0 # via pytest, tox
32+
pyaml==20.4.0 # via -r requirements.txt, scikit-optimize
3133
pyjwt==1.7.1 # via dohq-artifactory
3234
pyparsing==2.4.7 # via -r requirements.txt, matplotlib, packaging
33-
pytest-cov==2.10.0 # via -r dev-requirements.in
34-
pytest==5.4.3 # via -r dev-requirements.in, pytest-cov
35+
pytest-cov==2.10.1 # via -r dev-requirements.in
36+
pytest==6.0.1 # via -r dev-requirements.in, pytest-cov
3537
python-dateutil==2.8.1 # via -r requirements.txt, dohq-artifactory, matplotlib, pandas
3638
pytz==2020.1 # via -r requirements.txt, pandas
37-
pyyaml==5.3.1 # via -r requirements.txt, pre-commit
39+
pyyaml==5.3.1 # via -r requirements.txt, pre-commit, pyaml
3840
requests==2.24.0 # via dohq-artifactory
39-
scikit-learn==0.23.1 # via -r requirements.txt
40-
scipy==1.5.1 # via -r requirements.txt, scikit-learn
41+
scikit-learn==0.23.1 # via -r requirements.txt, scikit-optimize
42+
scikit-optimize==0.7.4 # via -r requirements.txt
43+
scipy==1.5.2 # via -r requirements.txt, scikit-learn, scikit-optimize
4144
six==1.15.0 # via -r requirements.txt, cycler, packaging, python-dateutil, tox, virtualenv
42-
sqlalchemy==1.3.18 # via -r requirements.txt
45+
sqlalchemy==1.3.19 # via -r requirements.txt
4346
threadpoolctl==2.1.0 # via -r requirements.txt, scikit-learn
44-
toml==0.10.1 # via pre-commit, tox
45-
tox==3.18.0 # via -r dev-requirements.in
46-
urllib3==1.25.9 # via requests
47-
virtualenv==20.0.27 # via pre-commit, tox
48-
wcwidth==0.2.5 # via pytest
47+
toml==0.10.1 # via pre-commit, pytest, tox
48+
tox==3.19.0 # via -r dev-requirements.in
49+
urllib3==1.25.10 # via requests
50+
virtualenv==20.0.28 # via pre-commit, tox

‎docs/baseclass.rst‎

Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ We then simply wrap a :class:`~sklearn.linear_model.LinearRegression` using our
2727
>>> bostondata = load_demo_dataset("boston")
2828
>>> # Remember to setup a train test split!
2929
>>> bostondata.create_train_test()
30-
<DemoData - Dataset>
30+
<BostonData - Dataset>
3131

3232
Creating your model
3333
~~~~~~~~~~~~~~~~~~~
@@ -205,21 +205,57 @@ search of the parameter space with a randomized search.
205205
.. doctest::
206206

207207
>>> from sklearn.ensemble import RandomForestRegressor
208-
>>> from scipy.stats import loguniform
208+
>>> from ml_tooling.search import Real
209209
>>> rand_forest = Model(RandomForestRegressor())
210210
>>>
211211
>>> search_space = {
212212
... "max_depth": [1, 3],
213-
... "min_weight_fraction_leaf": loguniform(1e-4, 1e0),
213+
... "min_weight_fraction_leaf": Real(0, 0.5),
214214
... }
215215
>>> best_estimator, results = rand_forest.randomsearch(bostondata, search_space, n_iter=2)
216216
>>> results #doctest:+SKIP
217217
ResultGroup(results=[<Result RandomForestRegressor: {'r2': 0.83}>, <Result RandomForestRegressor: {'r2': 0.56}>])
218218

219219
Here we specify the number of iterations `n_iter=2` just for demonstration purposes,
220-
n_iter is the number of points in the parameter samples that are tried out from the `sampler <https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterSampler.html>`_.
220+
n_iter is the number of times we sample from the parameter space to try. ML-Tooling uses skopt's :ref:`Spaces <skopt:space>`
221+
to define a sampling space. You can import them from :mod:`ml_tooling.search` or from skopt directly.
222+
221223
When a list is given in the search space, a linear distribution is used by default, but you may also
222-
pass other `distributions <https://docs.scipy.org/doc/scipy/reference/stats.html#continuous-distributions>`_
224+
pass other distributions. ML-Tooling supports :class:`~skopt.space.space.Real`, :class:`~skopt.space.space.Integer`
225+
and :class:`~skopt.space.space.Categorical`. Each of these also support prior distributions, if more granular
226+
distributions are required.
227+
228+
Performing a Bayesian Search
229+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
230+
ML-Tooling also supports Bayesian search - a stepwise search, where we build a surrogate model to estimate the effect
231+
of changing a given hyperparameter on the error. This surrogate model allows us to take steps in directions where the
232+
model thinks it can improve the error. Bayesian search is implemented using skopt and is a drop-in replacement for
233+
:meth:`~Model.randomsearch`.
234+
235+
.. doctest::
236+
237+
>>> from sklearn.ensemble import RandomForestRegressor
238+
>>> from ml_tooling.search import Real
239+
>>> rand_forest = Model(RandomForestRegressor())
240+
>>>
241+
>>> search_space = {
242+
... "max_depth": [1, 3],
243+
... "min_weight_fraction_leaf": Real(0, 0.5),
244+
... }
245+
>>> best_estimator, results = rand_forest.bayesiansearch(bostondata, search_space, n_iter=2)
246+
>>> results #doctest:+SKIP
247+
ResultGroup(results=[<Result RandomForestRegressor: {'r2': 0.83}>, <Result RandomForestRegressor: {'r2': 0.56}>])
248+
>>> from ml_tooling.search import Real
249+
>>> rand_forest = Model(RandomForestRegressor())
250+
>>>
251+
>>> search_space = {
252+
... "max_depth": [1, 3],
253+
... "min_weight_fraction_leaf": Real(0, 0.5),
254+
... }
255+
>>> best_estimator, results = rand_forest.bayesiansearch(bostondata, search_space, n_iter=2)
256+
>>> results #doctest:+SKIP
257+
ResultGroup(results=[<Result RandomForestRegressor: {'r2': 0.83}>, <Result RandomForestRegressor: {'r2': 0.56}>])
258+
223259

224260
Storage
225261
~~~~~~~
@@ -248,7 +284,7 @@ We can also load the model from a storage by specifying the filename to load in
248284

249285
.. doctest::
250286

251-
>>> loaded_linear = linear.load_estimator(storage, estimator_path.name)
287+
>>> loaded_linear = linear.load_estimator(estimator_path.name, storage=storage)
252288
>>> loaded_linear
253289
<Model: LinearRegression>
254290

@@ -281,6 +317,8 @@ Now users of your model package can always find your estimator through
281317
282318
>>> model.load_production_estimator('your_module_name')
283319
320+
By default, if no storage is specified, ML-Tooling will save models in your current working directory in a folder
321+
called `estimators`
284322

285323
Configuration
286324
~~~~~~~~~~~~~

‎docs/conf.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
intersphinx_mapping = {
5050
"pd": ("https://pandas.pydata.org/pandas-docs/stable/", None),
5151
"sklearn": ("https://scikit-learn.org/stable/", None),
52+
"skopt": ("https://scikit-optimize.github.io/stable/", None),
5253
}
5354

5455

‎docs/dataset.rst‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ Creating a FileDataset from a csv file
4343
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4444

4545
When we create a FileDataset, we need to specify the location of our datafiles -
46-
this will be available in the :code:`self.file_path` attribute
46+
this will be available in the :code:`self.file_path` attribute. ML Tooling can
4747
A more elaborate example of using this dataset can be found at :file:`../notebooks/Titanic Demo.ipynb`.
4848

4949
.. code-block::
@@ -53,12 +53,12 @@ A more elaborate example of using this dataset can be found at :file:`../noteboo
5353
>>>
5454
>>> class TitanicData(FileDataset):
5555
... def load_training_data(self):
56-
... data = pd.read_csv(self.file_path)
57-
... return data.drop('Survived', axis=1), data.Survived
56+
... data = self.read_file()
57+
... return data.drop(columns='Survived'), data.Survived
5858
...
59-
... def load_prediction_data(self):
60-
... data = pd.read_csv(self.file_path)
61-
... return data
59+
... def load_prediction_data(self, idx):
60+
... data = self.read_file()
61+
... return data.drop(columns="Survived").idx[[idx]]
6262
>>>
6363
>>> TitanicData("./titanic.csv")
6464
<TitanicData - FileDataset>
@@ -101,4 +101,4 @@ If you want to test your model on a demo datasets from :ref:`sklearn:datasets`,
101101
>>> bostondata = load_demo_dataset("boston")
102102
>>> # Remember to setup a train test split!
103103
>>> bostondata.create_train_test()
104-
<DemoData - Dataset>
104+
<BostonData - Dataset>

‎docs/plotting.rst‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,13 @@ To use the visualizations, access them using the `.plot` accessor on the
2323

2424
>>> result.plot.feature_importance()
2525

26+
.. code-block::
27+
28+
>>> result.plot.permutation_importance()
29+
30+
.. plot::
31+
32+
>>> result.plot.permutation_importance()
2633

2734
.. code-block:: python
2835
@@ -73,6 +80,12 @@ as arguments:
7380
Available Base plots
7481
--------------------
7582
- :meth:`~RegressionVisualize.feature_importance`
83+
Uses the estimator's learned coefficients or learned feature importance in the case of RandomForest
84+
to plot the relative importance of each feature. Note that for most usecases, permutation importance is
85+
going to be more accurate, but is also more computationally expensive.
86+
Pass a `class_index` parameter to select which class to plot for in a multi-class setting
87+
88+
- :meth:`~RegressionVisualize.permutation_importance`
7689
Uses random permutation to calculate feature importance by randomly permuting each column
7790
and measuring the difference in the model metric against the baseline.
7891

@@ -90,16 +103,20 @@ Available Classifier plots
90103
- :meth:`~ClassificationVisualize.roc_curve`
91104
Visualize a ROC curve for a classification model.
92105
Shows the relationship between the True Positive Rate and the False Positive Rate.
106+
Supports multi-class classification problems
93107

94108
- :meth:`~ClassificationVisualize.confusion_matrix`:
95109
Visualize a confusion matrix for a classification model.
96110
Shows the distribution of predicted labels vs actual labels
111+
Supports multi-class classification problems
97112

98113
- :meth:`~ClassificationVisualize.lift_curve`
99-
Visualizes how much of the target class we capture by setting different thresholds for probability
114+
Visualizes how much of the target class we capture by setting different thresholds for probability.
115+
Supports multi-class classification problems
100116

101-
- :meth:`~ClassificationVisualize.pr_curve`
117+
- :meth:`~ClassificationVisualize.precision_recall_curve`
102118
Visualize a Precision-Recall curve for a classification estimator. Estimator must implement a `predict_proba` method.
119+
Supports multi-class classification problems
103120

104121
Available Regression Plots
105122
--------------------------

‎docs/quickstart.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ We can save and load our model:
8686
>>> from ml_tooling.storage import FileStorage
8787
>>> storage = FileStorage('./estimator_dir')
8888
>>> file_path = regression.save_estimator(storage)
89-
>>> my_new_model = regression.load_estimator(storage, file_path.name)
89+
>>> my_new_model = regression.load_estimator(file_path.name, storage=storage)
9090
>>> my_new_model
9191
<Model: LinearRegression>
9292

0 commit comments

Comments
 (0)