Typed list delete slice by esc · Pull Request #1 · esc/numba

esc · 2019-06-25T13:30:54Z

This is the function numba_list_delete_slice which allows for mass removal of items. The code is quite involved, in fact it is probably the most complex function for the typed list to date. The code itself was largely cobbled together from the cpython implementation of the list.

esc · 2019-06-26T05:19:05Z

No review comments yet, will rebase to update.

This implements one of the two remaining C functions: numba_list_delete_slice at the C, compiler, and interpreter level. Tests are included. The implementation was copied from the relevant parts of the cpython source code in `list_ass_subscript` but doesn't exactly mirror any specific function. This should also enable acceleration of some other bits of the code.

When using setitem with a slice and an item that is smaller than the slice, we need to remove items. Until now, this was done with pop, which actually incurred a memory move of the tail of the list for every item being removed. Now that we can mass delete items in the list with `del` and slices, we used that functionality instead.

stuartarchibald · 2019-06-28T08:20:40Z

+ *      step != 0 and no Python negative indexing allowed.
+ *
+ * This code was copied and edited from the relevant section in
+ * list_ass_subscript from the cpython implementation.


Please add exact URL with SHA pointing to source, thanks.

The exact sources are listed at the top of the file, in the interest of deduplication, I added a comment pointing the reader to the top of the file.

esc · 2019-06-29T21:27:43Z

As discussed, will merge this PR, so that it can be tested with the rest of the typed-list implementation.

`_list_get_data` was moved and renamed to `_container_get_data`, this bug stems from a failed rebase.

Fix test_function_incompatible_templates test

Merge changes from origin repo

merg

Sync with original

More constant inference for 6822.

Advanced Indexing Support #1

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 (gdb) f 9 closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 4438 <string>: No such file or directory. (gdb) i args sched = {meminfo = 0x0, parent = 0x0, nitems = 0, itemsize = 0, data = 0x0, shape = {0}, strides = {0}} closure____locals______listcomp____v15____v2build__list__0 = {meminfo = 0x0, parent = 0x0} (gdb) f 8 498 mi->data = NRT_Reallocate(mi->data, size); (gdb) i args mi = 0x21d1660 size = 2510933856 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

This fixes a test, where a non-thread safe container is written to during testing To reproduce on at least `linux-64` and `osx-arm64` (and probably others): ``` NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting ``` On `linux-64`, this can be debugged with `gdb`: ``` (gdb) bt 0 0x00007ffff7c8018b in raise () from /lib/x86_64-linux-gnu/libc.so.6 1 0x00007ffff7c5f859 in abort () from /lib/x86_64-linux-gnu/libc.so.6 2 0x00007ffff7cca3ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 3 0x00007ffff7cd247c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 4 0x00007ffff7cd412c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 5 0x00007ffff7cd6105 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 6 0x00007ffff7cd82d6 in realloc () from /lib/x86_64-linux-gnu/libc.so.6 7 0x00007fffe706de21 in NRT_Reallocate (ptr=0x224aef0, size=2552489952) at numba/core/runtime/nrt.cpp:539 8 0x00007fffe706dcf6 in NRT_MemInfo_varsize_realloc (mi=0x1fe9580, size=2552489952) at numba/core/runtime/nrt.cpp:498 9 0x00007fffdcabde0d in _3cdynamic_3e::__numba_parfor_gufunc_0x7fffdc60eb20[abi:v19][abi:c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoA](Array<unsigned long long, 1, C, mutable, aligned>, list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e) (sched=..., closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438 10 0x00007fffdcab624e in __gufunc__._ZN13_3cdynamic_3e36__numba_parfor_gufunc_0x7fffdc60eb20B3v19B120c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoAE5ArrayIyLi1E1C7mutable7alignedE119list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e () 11 0x00007fffdc926a3b in thread_worker (arg=0x1bd49c0) at numba/np/ufunc/workqueue.c:567 12 0x00007ffff7f8f609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 13 0x00007ffff7d5c293 in clone () from /lib/x86_64-linux-gnu/libc.so.6 ``` On `osx-arm64` we can use `lldb`: ``` (lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64) Parallel: 0. Serial: 1 python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2) python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug Process 13575 stopped * thread numba#18, stop reason = signal SIGABRT frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 libsystem_kernel.dylib`: -> 0x19c35c704 <+8>: b.lo 0x19c35c724 ; <+40> 0x19c35c708 <+12>: pacibsp 0x19c35c70c <+16>: stp x29, x30, [sp, #-0x10]! 0x19c35c710 <+20>: mov x29, sp Target 0: (python3) stopped. (lldb) bt * thread numba#18, stop reason = signal SIGABRT * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180 frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908 frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104 frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628 frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356 frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112 frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388 frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60 frame numba#10: 0x000000013d4f41e0 frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148 ```

Type system implementation #1: Added initial implementation for a new type system using redundancies.

esc mentioned this pull request Jun 25, 2019

typed-list numba/numba#4151

Merged

esc force-pushed the typed_list_delete_slice branch from f608732 to 5fe1e3c Compare June 26, 2019 05:27

esc added 2 commits June 27, 2019 14:42

esc force-pushed the typed_list_delete_slice branch from 5fe1e3c to 261dcd1 Compare June 27, 2019 12:42

stuartarchibald reviewed Jun 28, 2019

View reviewed changes

add missing references to exact source

a767ce9

fix refactoring bug

488d6c0

`_list_get_data` was moved and renamed to `_container_get_data`, this bug stems from a failed rebase.

esc merged commit 5d20844 into typed_list Jun 29, 2019

esc deleted the typed_list_delete_slice branch June 29, 2019 23:36

esc pushed a commit that referenced this pull request Sep 11, 2019

Merge pull request #1 from stuartarchibald/pr_4345_cont

fa17d7c

Fix test_function_incompatible_templates test

esc pushed a commit that referenced this pull request Oct 1, 2019

Merge pull request #1 from numba/master

3d50c71

Merge changes from origin repo

esc pushed a commit that referenced this pull request Oct 8, 2020

Applying review comments #1

d8c0e37

esc pushed a commit that referenced this pull request Oct 8, 2020

Merge pull request #1 from numba/master

8bc0695

merg

esc pushed a commit that referenced this pull request Oct 14, 2020

Applying review comments #1

55e5f79

esc pushed a commit that referenced this pull request Nov 16, 2020

Merge pull request #1 from numba/master

e464068

Sync with original

esc pushed a commit that referenced this pull request Mar 26, 2021

Merge pull request #1 from stuartarchibald/pr_6822_continued

700c0b3

More constant inference for 6822.

esc pushed a commit that referenced this pull request Nov 23, 2021

Applying review comments #1

ae84006

esc pushed a commit that referenced this pull request Mar 31, 2023

Merge pull request numba#8238 from kc611/adv_idx_1

26bc501

Advanced Indexing Support #1

esc pushed a commit that referenced this pull request Aug 14, 2024

Merge pull request numba#9662 from kc611/new_ty_sys

2001717

Type system implementation #1: Added initial implementation for a new type system using redundancies.

esc pushed a commit that referenced this pull request Aug 14, 2024

Merge pull request numba#9686 from kc611/new_ty_sys_review

e980eba

Type system implementation #1: Added initial implementation for a new type system using redundancies.

Navigation Menu

Repository navigation

Typed list delete slice - #1

Typed list delete slice#1
esc merged 4 commits into
typed_listfrom
typed_list_delete_slice

esc commented Jun 25, 2019

Uh oh!

esc commented Jun 26, 2019

Uh oh!

stuartarchibald Jun 28, 2019

Uh oh!

esc Jun 29, 2019

Uh oh!

esc commented Jun 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

esc commented Jun 25, 2019

Uh oh!

esc commented Jun 26, 2019

Uh oh!

stuartarchibald Jun 28, 2019

Choose a reason for hiding this comment

Uh oh!

esc Jun 29, 2019

Choose a reason for hiding this comment

Uh oh!

esc commented Jun 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants