Skip to content
This repository was archived by the owner on Feb 2, 2024. It is now read-only.

Commit 43f3c88

Browse files
Vyacheslav-Smirnovshssf
Vyacheslav-Smirnov
authored and
shssf
committed
Change tests execution (#111)
* Remove spark dependency (#102) Remove spark dependency from HPA; use pre-generated sdf_dt.pq * explicitly adding data-file (#104) * HPAT Build: Code style check for C and Python sources (#103) * HPAT Build: Code style check for C and Python sources * PR103. Comments partially addressed * Code style change part 1 (#106) * Style check config fo pystyle (#105) * Fix for pandas.merge wrong overload handling of 'on' args (#99) Problem description: merge_overload and merge_asof_overload functions use 'on' argument value to compute 'left_on' and 'right_on' arguments in a way that breaks type stability, causing compilation failure when 'on' is assigned a StringLiteral value. Error: File "../hpat/hiframes/dataframe_pass.py", line 202, in _run_assign return self._run_call(assign, lhs, rhs) File "../hpat/hiframes/dataframe_pass.py", line 522, in _run_call return self._run_call_join(assign, lhs, rhs) File "../hpat/hiframes/dataframe_pass.py", line 1488, in _run_call_join left_on = self._get_const_or_list(left_on_var) File "../hpat/hiframes/dataframe_pass.py", line 2135, in _get_const_or_list raise ValueError(err_msg) ValueError: Failed in hpat mode pipeline (step: typed dataframe pass) None Following tests should be fixed with this commit: test_join_cat1 (hpat.tests.test_join.TestJoin) test_join_cat2 (hpat.tests.test_join.TestJoin) test_join_cat_parallel1 (hpat.tests.test_join.TestJoin) test_join_datetime_seq1 (hpat.tests.test_join.TestJoin) test_join_left_seq1 (hpat.tests.test_join.TestJoin) test_join_left_seq2 (hpat.tests.test_join.TestJoin) test_join_outer_seq1 (hpat.tests.test_join.TestJoin) test_join_right_seq1 (hpat.tests.test_join.TestJoin) test_merge_asof_seq1 (hpat.tests.test_join.TestJoin) * [STL] PEP8 code style for 'test_strings.py', 'test_utils.py', 'test_series.py' (#97) * pep8 style for 'test_strings.py'; flake8 check successful * pep8 style for 'test_utils.py' * pep8 style for 'test_series.py'; more readable * fixed 'test_string_series' * removed extra white spaces * deleted mention of flake8 * trigger build * Code style change part 2 (#107) * code_style_change_part_2 * Add more check in style configuration (#108) * code_style_part_3 (#109) * Fix boost runtime issue on Ubuntu16.04 with gcc 5.4 (#92) * Code style change part 4 (#110) * Cahnge tests execution Actually test suite should be executed via hpat.runtests: python -u -m hpat.runtests -v This resolve the issue with doulbe test suite execution which occurs due to the "python -u -m unittest -v" command import all files in tree including runtests.py and runtests.py triggers 1-st suite execution. Then unittest triggers 2-d. Add decorator to execute some tests (mostly parallel) 2 or more times (depending on existance of REPEAT_TEST_NUMBER environment variable) This should highlight issues like the test fails if executed twice because is corrupts memory during first execution (like test_series_head_index_parallel1) Skip test_series_head_index_parallel1 because it triggers memory corruption. This should be fixed. * Revert "Code style change part 4 (#110)" This reverts commit dfc54ee. * Revert "Fix boost runtime issue on Ubuntu16.04 with gcc 5.4 (#92)" This reverts commit 231a76c. * Revert "code_style_part_3 (#109)" This reverts commit 4070ce3. * Revert "Add more check in style configuration (#108)" This reverts commit abf5bd0. * Revert "Code style change part 2 (#107)" This reverts commit 9076493. * Revert "[STL] PEP8 code style for 'test_strings.py', 'test_utils.py', 'test_series.py' (#97)" This reverts commit 8641f7a. * Revert "Fix for pandas.merge wrong overload handling of 'on' args (#99)" This reverts commit a2a8ee5. * Revert "Style check config fo pystyle (#105)" This reverts commit 551c0e3. * Revert "Code style change part 1 (#106)" This reverts commit 6dae0b3. * Revert "HPAT Build: Code style check for C and Python sources (#103)" This reverts commit 1a30e4f. * Revert "explicitly adding data-file (#104)" This reverts commit 34a2260. * Revert "Remove spark dependency (#102)" This reverts commit 9e77fde. * Wrap functions to be executed twice in runtests.py * Update runtests.py Execute every test specified times, which is set via the REPEAT_TEST_NUMBER environment variable. Skip test_series_list_str_unbox1 because is fails on the second launch with Segmentation fault * Apply comments from review Rename REPEAT_TEST_NUMBER to HPAT_REPEAT_TEST_NUMBER Use os.getenv to get value for HPAT_REPEAT_TEST_NUMBER
1 parent e1ba769 commit 43f3c88

3 files changed

Lines changed: 29 additions & 5 deletions

File tree

‎buildscripts/test.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@
33
source activate $CONDA_ENV
44

55
# generate test data for test_io
6-
python hpat/tests/gen_test_data.py
6+
python -m hpat.tests.gen_test_data
77

88
if [ "$RUN_COVERAGE" == "yes" ]; then
99
export PYTHONPATH=.
1010
coverage erase
1111
coverage run --source=./hpat --omit ./hpat/ml/*,./hpat/xenon_ext.py,./hpat/ros.py,./hpat/cv_ext.py,./hpat/tests/* -m unittest
1212
else
13-
mpiexec -n $NUM_PES python -u -m unittest -v
13+
mpiexec -n $NUM_PES python -u -m hpat.runtests -v
1414
fi

‎hpat/runtests.py‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,33 @@
1+
import os
12
import unittest
23
import hpat.tests
34

5+
"""
6+
Every test in suite can be executed specified times using
7+
desired value for HPAT_REPEAT_TEST_NUMBER environment variable.
8+
This can be used to locate scpecific failures occured
9+
on next execution of affected test.
410
11+
loadTestsFromModule returns TestSuite obj with _tests member
12+
which contains further TestSuite instanses for each found testCase:
13+
hpat_tests = TestSuite(hpat.tests)
14+
TestSuite(hpat.tests)._tests = [TestSuite(hpat.tests.TestBasic), TestSuite(hpat.tests.TestDataFrame), ...]
15+
TestSuite(hpat.tests.TestBasic)._tests = [TestBasic testMethod=test_array_reduce, ...]
16+
"""
517
def load_tests(loader, tests, pattern):
618
suite = unittest.TestSuite()
7-
suite.addTests(loader.loadTestsFromModule(hpat.tests))
19+
hpat_tests = loader.loadTestsFromModule(hpat.tests)
20+
repeat_test_number = int(os.getenv('HPAT_REPEAT_TEST_NUMBER', '1'))
21+
22+
if repeat_test_number > 1:
23+
for i, test_case in enumerate(hpat_tests):
24+
extended_tests = []
25+
for test in test_case:
26+
for _ in range(repeat_test_number):
27+
extended_tests.append(test)
28+
hpat_tests._tests[i]._tests = extended_tests
29+
30+
suite.addTests(hpat_tests)
831
return suite
932

1033

‎hpat/tests/test_series.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -591,6 +591,7 @@ def test_impl():
591591

592592
pd.testing.assert_series_equal(hpat_func(), test_impl())
593593

594+
@unittest.skip("ERROR: Segmentation fault on the second launch (using HPAT_REPEAT_TEST_NUMBER=2)")
594595
def test_series_list_str_unbox1(self):
595596
def test_impl(A):
596597
return A.iloc[0]
@@ -1314,9 +1315,9 @@ def test_impl():
13141315
Exact errors:
13151316
1. Segmentation fault in TestBasic.test_rebalance
13161317
2. FAIL in TestBasic.test_astype with following error message:
1317-
test_astype (hpat.tests.test_basic.TestBasic) ...
1318+
test_astype (hpat.tests.test_basic.TestBasic) ...
13181319
Fatal error in MPI_Allreduce: Message truncated, error stack:
1319-
MPI_Allreduce(907)..................: MPI_Allreduce(sbuf=0x7ffe3b734128, rbuf=0x7ffe3b734120, count=1,
1320+
MPI_Allreduce(907)..................: MPI_Allreduce(sbuf=0x7ffe3b734128, rbuf=0x7ffe3b734120, count=1,
13201321
MPI_LONG_LONG_INT, MPI_SUM, MPI_COMM_WORLD) failed
13211322
MPIR_Allreduce_impl(764)............:
13221323
MPIR_Allreduce_intra(238)...........:

0 commit comments

Comments
 (0)