Repository navigation
Make always_run test decorator a tag and improve shard tests. - #8675
Merged
Merged
Conversation
stuartarchibald
commented
Dec 16, 2022
Comment on lines
+169
to
+187
| # check that the always running tests are in every shard, and then | ||
| # remove them from the shards | ||
| for shard in sharded_sets: | ||
| for test in always_running: | ||
| self.assertIn(test, shard) | ||
| shard.remove(test) | ||
| self.assertNotIn(test, shard) | ||
|
|
||
| # check that there is no overlap between the shards | ||
| for a, b in itertools.combinations(sharded_sets, 2): | ||
| self.assertFalse(a & b) | ||
|
|
||
| # check that the sum of the shards and the always running tests is the | ||
| # same as the full listing | ||
|
|
||
| sum_of_parts = set() | ||
| [sum_of_parts.update(x) for x in sharded_sets] | ||
| sum_of_parts.update(always_running) | ||
|
|
||
| full_listing = set(self._get_numba_tests_from_listing( | ||
| self.get_testsuite_listing([]))) | ||
|
|
||
| self.assertEqual(sum_of_parts, full_listing) |
Contributor
Author
There was a problem hiding this comment.
This is quite verbose but with the intention of being clear. Another option might be to just compute the intersection of each combination of shards and make sure that it matches the set of "always_running" tests, then combine all shards and seeing if they match the full listing.
stuartarchibald
commented
Dec 16, 2022
This patch: * Makes an `always_run` test decorator to use on tests that should run in all test shards, it's based on the existing test `tag` mechanism. * Updates the test runner to recognise the `always_run` tag when sharding so as to ensure that tests tagged with this tag will always be included in a shard. * Updates the unit tests to ensure fundamental properties with regards to sharding. This includes shard uniqueness, the presence of the `always_run` tests in shards and also that the sum of the test shards and `always_run` tests is the same as the full test listing. * Fixes up a few references slice->shard. Fixes numba#8657
stuartarchibald
force-pushed
the
fix/8657
branch
from
December 16, 2022 17:03
82a757a to
8d5b70b
Compare
apmasell
reviewed
Jan 10, 2023
Resolved conflicts in: buildscripts/azure/azure-windows.yml
As title.
Contributor
Author
stuartarchibald
marked this pull request as ready for review
January 11, 2023 10:10
stuartarchibald
added
4 - Waiting on reviewer
As title. Other testing in this module asserts that the shards are of a suitable size.
apmasell
approved these changes
Jan 11, 2023
stuartarchibald
added
4 - Waiting on second reviewer
sklam
approved these changes
Jan 11, 2023
sklam
added
5 - Ready to merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch:
always_runtest decorator to use on tests that should run in all test shards, it's based on the existing testtagmechanism.always_runtag when sharding so as to ensure that tests tagged with this tag will always be included in a shard.always_runtests in shards and also that the sum of the test shards andalways_runtests is the same as the full test listing.Fixes #8657