Skip to content

Commit f579b70

Browse files
JuanVqzJuanVqz
authored
Merge pull request #243 from fastruby/feature/benchmark-claims
Name each benchmark's winner and lint it
2 parents 6e6e567 + c50ae44 commit f579b70

46 files changed

Lines changed: 481 additions & 329 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/scripts/lint-benchmarks.rb‎

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,21 @@
66
# so every file uses the default
77
# - no Benchmark.ips sits inside a method that never runs from the top of the file,
88
# which would benchmark nothing at all
9+
# - every Benchmark.ips block states which report should win: exactly one
10+
# report calls `fastest` (else `faster`, else `fast`), and it comes first
911
#
1012
# Usage: ruby .github/scripts/lint-benchmarks.rb [files...] (needs Ruby 3.3+)
1113
require "prism"
1214

13-
# Calls named `name` that have a block, not looking inside the ones found.
14-
def find_calls(node, name, found = [])
15+
# Calls named `name`, not looking inside the ones found. Only calls with a
16+
# block, unless `with_block: false` (reports can be a code string: x.report(label, code)).
17+
def find_calls(node, name, with_block: true, found: [])
1518
return found unless node
1619

17-
if node.is_a?(Prism::CallNode) && node.name == name && node.block
20+
if node.is_a?(Prism::CallNode) && node.name == name && (node.block || !with_block)
1821
found << node
1922
else
20-
node.compact_child_nodes.each { |child| find_calls(child, name, found) }
23+
node.compact_child_nodes.each { |child| find_calls(child, name, with_block: with_block, found: found) }
2124
end
2225
found
2326
end
@@ -29,6 +32,28 @@ def any_call?(node, &test)
2932
node.compact_child_nodes.any? { |child| any_call?(child, &test) }
3033
end
3134

35+
CLAIM_METHODS = %i[fastest faster fast].freeze
36+
37+
# Method names called under `node` without a receiver.
38+
def called_names(node, names = [])
39+
return names unless node
40+
41+
names << node.name if node.is_a?(Prism::CallNode) && node.receiver.nil?
42+
node.compact_child_nodes.each { |child| called_names(child, names) }
43+
names
44+
end
45+
46+
def claim_problem(ips)
47+
reports = find_calls(ips.block, :report, with_block: false).map { |r| called_names(r.block) }
48+
top = CLAIM_METHODS.find { |name| reports.any? { |calls| calls.include?(name) } }
49+
return "no claim. Wrap each report in a method and name the winner's `fast` (or `faster`, `fastest`)" unless top
50+
51+
claimed = reports.count { |calls| calls.include?(top) }
52+
return "#{claimed} reports call `#{top}`. Name only the winner `#{top}`" if claimed > 1
53+
54+
"the report calling `#{top}` must come first" unless reports.first.include?(top)
55+
end
56+
3257
TIMING_KEYS = %w[time warmup].freeze
3358

3459
# x.time = 20, x.warmup = 5, or x.config with a time or warmup key,
@@ -159,6 +184,9 @@ def lint(file)
159184
name = owner.keys.last.delete_prefix("#")
160185
problems << "#{where}: inside `def #{name}`, which never runs from the top of the file"
161186
end
187+
188+
claim = claim_problem(ips)
189+
problems << "#{where}: #{claim}" if claim
162190
end
163191
problems
164192
end

‎CONTRIBUTING.md‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,37 @@ Keep that shape: end every `Benchmark.ips` block with `x.compare!`, keep the
3939
default timing (no `Benchmark.ips(20)`, `x.time = ...` or `x.config(time: ...)`),
4040
so every entry is measured the same way, and make sure
4141
the file actually calls `Benchmark.ips` when it runs
42-
(not only inside a method nothing calls). CI checks all three.
42+
(not only inside a method nothing calls). CI checks these, and the naming below.
43+
44+
The method names say which report should win.
45+
CI checks that exactly one report is named the winner and that it comes first.
46+
Wrap every report in a method, so they all pay the same call cost.
47+
Name them by rank and list the winner first.
48+
The names grow outward from the line between fast and slow: the recommended side uses `fast`, then `faster`, then `fastest`; the other side uses `slow`, then `slower`, then `slowest`.
49+
The names are relative: `slow` only means slower than `fast`.
50+
When a ranking could look odd, say why in one line, like in `code/array/length-vs-size-vs-count.rb`:
51+
52+
```ruby
53+
def faster
54+
ARRAY.length
55+
end
56+
57+
# Array#size is an alias of Array#length, so these two should tie.
58+
def fast
59+
ARRAY.size
60+
end
61+
62+
def slow
63+
ARRAY.count
64+
end
65+
66+
Benchmark.ips do |x|
67+
x.report("Array#length") { faster }
68+
x.report("Array#size") { fast }
69+
x.report("Array#count") { slow }
70+
x.compare!
71+
end
72+
```
4373

4474
Run your result:
4575

‎README.md‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,7 @@ using an if statement: 15517955.2 i/s
134134
String#constantize: 10556362.4 i/s - 1.47x slower
135135
```
136136

137-
##### `raise` vs `E2MM#Raise` for raising (and defining) exceptions [code](code/general/raise-vs-e2mmap.rb)
137+
##### `raise` vs `E2MM#Raise` for raising (and defining) exceptions [code](code/general/raise-vs-e2mmap.rb) [custom exception code](code/general/raise-custom-vs-e2mmap.rb)
138138

139139
Ruby's [Exception2MessageMapper module](http://ruby-doc.org/stdlib-2.2.0/libdoc/e2mmap/rdoc/index.html) allows one to define and raise exceptions with predefined messages.
140140

@@ -155,7 +155,10 @@ Ruby exception: Kernel#raise
155155
Comparison:
156156
Ruby exception: Kernel#raise: 2570660.6 i/s
157157
Ruby exception: E2MM#Raise: 88268.9 i/s - 29.12x slower
158+
```
158159

160+
```
161+
$ ruby -v code/general/raise-custom-vs-e2mmap.rb
159162
ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [arm64-darwin24]
160163
Warming up --------------------------------------
161164
Custom exception: E2MM#Raise
@@ -1058,23 +1061,21 @@ Comparison:
10581061

10591062
```
10601063
$ ruby -v code/proc-and-block/proc-call-vs-yield.rb
1061-
ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [arm64-darwin24]
1064+
ruby 4.0.7 (2026-09-15 revision 229531a6cf) +PRISM [aarch64-linux]
10621065
Warming up --------------------------------------
1063-
block.call 2.261M i/100ms
1064-
block + yield 2.314M i/100ms
1065-
unused block 3.025M i/100ms
1066-
yield 2.971M i/100ms
1066+
yield 2.156M i/100ms
1067+
block + yield 1.605M i/100ms
1068+
block.call 1.609M i/100ms
10671069
Calculating -------------------------------------
1068-
block.call 22.057M (± 6.0%) i/s (45.34 ns/i) - 110.796M in 5.043129s
1069-
block + yield 23.280M (± 0.6%) i/s (42.96 ns/i) - 117.997M in 5.068779s
1070-
unused block 30.609M (± 1.3%) i/s (32.67 ns/i) - 154.268M in 5.040991s
1071-
yield 29.921M (± 0.6%) i/s (33.42 ns/i) - 151.512M in 5.063842s
1070+
yield 20.615M (±10.0%) i/s (48.51 ns/i) - 103.495M in 5.020450s
1071+
block + yield 16.824M (± 9.2%) i/s (59.44 ns/i) - 85.066M in 5.056288s
1072+
block.call 14.608M (±14.3%) i/s (68.46 ns/i) - 73.995M in 5.065432s
10721073
10731074
Comparison:
1074-
unused block: 30608512.5 i/s
1075-
yield: 29921356.8 i/s - 1.02x slower
1076-
block + yield: 23279981.0 i/s - 1.31x slower
1077-
block.call: 22056758.6 i/s - 1.39x slower
1075+
yield: 20614679.3 i/s
1076+
block + yield: 16823796.9 i/s - 1.23x slower
1077+
block.call: 14607819.3 i/s - 1.41x slower
1078+
10781079
```
10791080

10801081
### String

‎code/array/bsearch-vs-find.rb‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,17 @@
11
require 'benchmark/ips'
22

3-
data = [*0..100_000_000]
3+
NUMBERS = [*0..100_000_000]
4+
5+
def fast
6+
NUMBERS.bsearch { |number| number > 77_777_777 }
7+
end
8+
9+
def slow
10+
NUMBERS.find { |number| number > 77_777_777 }
11+
end
412

513
Benchmark.ips do |x|
6-
x.report('find') { data.find { |number| number > 77_777_777 } }
7-
x.report('bsearch') { data.bsearch { |number| number > 77_777_777 } }
14+
x.report('bsearch') { fast }
15+
x.report('find') { slow }
816
x.compare!
917
end

‎code/array/insert-vs-unshift.rb‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,17 @@
11
require 'benchmark/ips'
22

3-
Benchmark.ips do |x|
4-
x.report('Array#unshift') do
5-
array = []
6-
100_000.times { |i| array.unshift(i) }
7-
end
3+
def fast
4+
array = []
5+
100_000.times { |i| array.unshift(i) }
6+
end
87

9-
x.report('Array#insert') do
10-
array = []
11-
100_000.times { |i| array.insert(0, i) }
12-
end
8+
def slow
9+
array = []
10+
100_000.times { |i| array.insert(0, i) }
11+
end
1312

13+
Benchmark.ips do |x|
14+
x.report('Array#unshift') { fast }
15+
x.report('Array#insert') { slow }
1416
x.compare!
1517
end

‎code/array/length-vs-size-vs-count.rb‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,22 @@
22

33
ARRAY = [*1..100]
44

5+
def faster
6+
ARRAY.length
7+
end
8+
9+
# Array#size is an alias of Array#length, so these two should tie.
10+
def fast
11+
ARRAY.size
12+
end
13+
14+
def slow
15+
ARRAY.count
16+
end
17+
518
Benchmark.ips do |x|
6-
x.report("Array#length") { ARRAY.length }
7-
x.report("Array#size") { ARRAY.size }
8-
x.report("Array#count") { ARRAY.count }
19+
x.report("Array#length") { faster }
20+
x.report("Array#size") { fast }
21+
x.report("Array#count") { slow }
922
x.compare!
1023
end

‎code/array/shuffle-first-vs-sample.rb‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,16 @@
22

33
ARRAY = [*1..100]
44

5-
def slow
6-
ARRAY.shuffle.first
7-
end
8-
95
def fast
106
ARRAY.sample
117
end
128

9+
def slow
10+
ARRAY.shuffle.first
11+
end
12+
1313
Benchmark.ips do |x|
14-
x.report('Array#shuffle.first') { slow }
1514
x.report('Array#sample') { fast }
15+
x.report('Array#shuffle.first') { slow }
1616
x.compare!
1717
end

‎code/enumerable/each-push-vs-map.rb‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,17 @@
22

33
ARRAY = (1..100).to_a
44

5+
def fast
6+
ARRAY.map { |i| i }
7+
end
8+
59
def slow
610
array = []
711
ARRAY.each { |i| array.push i }
812
end
913

10-
def fast
11-
ARRAY.map { |i| i }
12-
end
13-
1414
Benchmark.ips do |x|
15-
x.report('Array#each + push') { slow }
1615
x.report('Array#map') { fast }
16+
x.report('Array#each + push') { slow }
1717
x.compare!
1818
end

‎code/enumerable/each-vs-for-loop.rb‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,20 +2,20 @@
22

33
ARRAY = [*1..100]
44

5-
def slow
6-
for number in ARRAY do
5+
def fast
6+
ARRAY.each do |number|
77
number
88
end
99
end
1010

11-
def fast
12-
ARRAY.each do |number|
11+
def slow
12+
for number in ARRAY do
1313
number
1414
end
1515
end
1616

1717
Benchmark.ips do |x|
18-
x.report('For loop') { slow }
1918
x.report('#each') { fast }
19+
x.report('For loop') { slow }
2020
x.compare!
2121
end

‎code/enumerable/inject-symbol-vs-block.rb‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
require "rubygems"
21
require "benchmark/ips"
32

43
ARRAY = (1..1000).to_a
54

6-
def fastest
5+
def faster
76
ARRAY.inject(:+)
87
end
98

9+
# Symbol#to_proc beats the block on plain CRuby up to 3.4; the block wins with YJIT or ZJIT, on 4.0 and newer, and on JRuby.
1010
def fast
1111
ARRAY.inject(&:+)
1212
end
@@ -16,9 +16,8 @@ def slow
1616
end
1717

1818
Benchmark.ips do |x|
19-
x.report('inject symbol') { fastest }
19+
x.report('inject symbol') { faster }
2020
x.report('inject to_proc') { fast }
2121
x.report('inject block') { slow }
22-
2322
x.compare!
2423
end

0 commit comments

Comments
 (0)