Skip to content
This repository was archived by the owner on Feb 2, 2024. It is now read-only.

Parallel isin #603

Merged
AlexanderKalistratov merged 31 commits into from
May 28, 2020
Merged

Parallel isin #603

Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
e3ea2a6
parallel and add mask for methods: capitalize, casefold, swapcase, title
Rubtsowa Feb 6, 2020
a30f1c4
correct master
Rubtsowa Feb 6, 2020
2730e63
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 10, 2020
c8a903f
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 10, 2020
4e3c3b0
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 11, 2020
c5f71ba
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 11, 2020
afdee1c
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 12, 2020
301a911
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 13, 2020
e8deb2f
add empty line
Rubtsowa Feb 13, 2020
37d2de5
Merge branch 'master' of https://github.com/IntelPython/hpat
Rubtsowa Feb 13, 2020
5b714c4
while not work
Rubtsowa Feb 13, 2020
24d9ab0
not scallable
Rubtsowa Feb 14, 2020
db68dd5
change
Rubtsowa Feb 14, 2020
2f0813b
add use numpy.empty
Rubtsowa Feb 14, 2020
704ea7d
use set
Rubtsowa Feb 17, 2020
928e9eb
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Feb 17, 2020
ab89197
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Feb 18, 2020
75fc11e
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Feb 27, 2020
fc4a934
work with str_list
Rubtsowa Feb 28, 2020
511e16c
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Feb 28, 2020
e5f5112
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Mar 3, 2020
e0cc9ae
change isinstance in Ser.isin
Rubtsowa Mar 3, 2020
1b4ebbb
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Mar 3, 2020
f1cef7b
add perf test for Series.isin with string input_data
Rubtsowa Mar 3, 2020
7fcef28
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa Mar 27, 2020
741e139
change doc
Rubtsowa Mar 27, 2020
112f9f5
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa May 7, 2020
58fedfa
change perf tests
Rubtsowa May 8, 2020
128ca8a
Merge branch 'master' of https://github.com/IntelPython/hpat into par…
Rubtsowa May 26, 2020
40b61b4
for on chunks
Rubtsowa May 26, 2020
e6c3519
return use prange
Rubtsowa May 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add use numpy.empty
  • Loading branch information
@Rubtsowa
Rubtsowa committed Feb 14, 2020
commit 2f0813b8d63149b695bdaa48e8f2ca20ada4c0e8
2 changes: 1 addition & 1 deletion sdc/datatypes/hpat_pandas_series_functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1992,7 +1992,7 @@ def hpat_pandas_series_isin_impl(self, values):
# return pandas.Series(np.isin(self._data, values))

data_len = len(self._data)
result = numpy.array([False] * data_len)
result = numpy.empty(data_len, dtype=numpy.bool_)
for i in numba.prange(data_len):
result[i] = self._data[i] in values

@densmirn densmirn Feb 14, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't you try to convert values to set?

@Rubtsowa Rubtsowa Feb 14, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This slows down the execution of the function, even on lists with large repetitions.

@AlexanderKalistratov AlexanderKalistratov Feb 14, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rubtsowa
no, it doesn't

name nthreads type size median min max compile boxing
Series.isin 1.0 Python 10000000.0 0.009159 0.009127 0.009223 NaN NaN
1.0 SDC 10000000.0 0.027854 0.027554 0.028204 0.104338 0.000232
8.0 Python 10000000.0 0.009403 0.008639 0.012322 NaN NaN
8.0 SDC 10000000.0 0.006398 0.006361 0.007950 0.106473 0.00027

Moreover, I would say, it speeds everything up

@AlexanderKalistratov AlexanderKalistratov Feb 14, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without set:

name nthreads type size median min max compile boxing
Series.isin 1.0 Python 10000000.0 0.008859 0.008839 0.008965 NaN NaN
1.0 SDC 10000000.0 0.127839 0.127763 0.127877 0.095031 0.000235
8.0 Python 10000000.0 0.009093 0.008691 0.012178 NaN NaN
8.0 SDC 10000000.0 0.324509 0.322576 0.454564 0.08109 0.000293

@AlexanderKalistratov AlexanderKalistratov Feb 14, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial version (list comprehension):

name nthreads type size median min max compile boxing
Series.isin 1.0 Python 10000000.0 0.005723 0.005515 0.006462 NaN NaN
1.0 SDC 10000000.0 0.252864 0.232145 0.253373 0.066061 0.000244
8.0 Python 10000000.0 0.005694 0.005566 0.006526 NaN NaN
8.0 SDC 10000000.0 0.232615 0.232179 0.232622 0.06533 0.000257

@Rubtsowa Rubtsowa Feb 17, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without set:
Ran 1 test in 0.505s

with set:
Ran 1 test in 1.676s

@densmirn densmirn Feb 17, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it time of execution of unit test or perf. test? Why do you use python.unittest time from command line?


Expand Down