Skip to content

Commit 9afe4ec

Browse files
pytorchbothuydhn
pytorchbot
and
huydhn
authored
Update torchbench model expected accuracy values after pinning numpy (#129986)
* Update torchbench model expected accuracy values after pinning numpy (#129213) After pinning numpy on torchbench, we need to move torchbench inductor benchmark jobs out of unstable state asap, so that more failures don't sneak it. I'm updating the expected values here to make trunk green. Pull Request resolved: #129213 Approved by: https://github.com/xuzhao9, https://github.com/malfet, https://github.com/desertfire (cherry picked from commit b72ef9d) * No change to yolov3 --------- Co-authored-by: Huy Do <huydhn@gmail.com>
1 parent 499621e commit 9afe4ec

13 files changed

Lines changed: 11 additions & 63 deletions

‎benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_inference.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,pass,12
18-
19-
20-
2117
LearningToPaint,pass,0
2218

2319

@@ -150,7 +146,7 @@ hf_Bert_large,pass,0
150146

151147

152148

153-
hf_BigBird,pass,46
149+
hf_BigBird,pass,0
154150

155151

156152

‎benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_training.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,eager_fail_to_run,0
18-
19-
20-
2117
LearningToPaint,pass,6
2218

2319

@@ -98,7 +94,7 @@ hf_Bert_large,pass,6
9894

9995

10096

101-
hf_BigBird,pass, 52
97+
hf_BigBird,pass,6
10298

10399

104100

‎benchmarks/dynamo/ci_expected_accuracy/aot_inductor_torchbench_inference.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,fail_to_run,0
18-
19-
20-
2117
LearningToPaint,pass,0
2218

2319

@@ -138,7 +134,7 @@ hf_Bert_large,pass,0
138134

139135

140136

141-
hf_BigBird,fail_to_run,0
137+
hf_BigBird,fail_accuracy,0
142138

143139

144140

‎benchmarks/dynamo/ci_expected_accuracy/cpu_inductor_torchbench_inference.csv‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,6 @@ Background_Matting,pass_due_to_skip,0
1010

1111

1212

13-
DALLE2_pytorch,model_fail_to_load,0
14-
15-
16-
1713
LearningToPaint,pass,0
1814

1915

‎benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_inference.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,pass,12
18-
19-
20-
2117
LearningToPaint,pass,0
2218

2319

@@ -150,7 +146,7 @@ hf_Bert_large,pass,0
150146

151147

152148

153-
hf_BigBird,pass,46
149+
hf_BigBird,fail_to_run,0
154150

155151

156152

‎benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_training.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,eager_fail_to_run,0
18-
19-
20-
2117
LearningToPaint,pass,6
2218

2319

@@ -98,7 +94,7 @@ hf_Bert_large,pass,6
9894

9995

10096

101-
hf_BigBird,pass,52
97+
hf_BigBird,fail_to_run,3
10298

10399

104100

‎benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_torchbench_inference.csv‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,6 @@ Background_Matting,pass_due_to_skip,0
1010

1111

1212

13-
DALLE2_pytorch,model_fail_to_load,0
14-
15-
16-
1713
LearningToPaint,pass,0
1814

1915

‎benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_inference.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,pass,12
18-
19-
20-
2117
LearningToPaint,pass,0
2218

2319

@@ -150,7 +146,7 @@ hf_Bert_large,pass,0
150146

151147

152148

153-
hf_BigBird,fail_accuracy,46
149+
hf_BigBird,fail_to_run,0
154150

155151

156152

‎benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,eager_fail_to_run,0
18-
19-
20-
2117
LearningToPaint,pass,6
2218

2319

@@ -98,7 +94,7 @@ hf_Bert_large,pass,6
9894

9995

10096

101-
hf_BigBird,pass,52
97+
hf_BigBird,fail_to_run,3
10298

10399

104100

‎benchmarks/dynamo/ci_expected_accuracy/dynamo_eager_torchbench_inference.csv‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,6 @@ Background_Matting,pass_due_to_skip,0
1414

1515

1616

17-
DALLE2_pytorch,pass,12
18-
19-
20-
2117
LearningToPaint,pass,0
2218

2319

@@ -150,7 +146,7 @@ hf_Bert_large,pass,0
150146

151147

152148

153-
hf_BigBird,pass,46
149+
hf_BigBird,pass,0
154150

155151

156152

0 commit comments

Comments
 (0)