I was working on laminas/laminas-servicemanager#93 today, and noticed that we still have the codebase sprinkled with blocks referencing #304
In fact, #304 has been closed as "old", but still applies today.
My assumption was that @BeforeMethods would be called before each call to said bench method: doesn't seem to be the case, and that leads to benchmarking warmed up caches too, which is a problem (especially if we're benchmarking said warmup).
In following example, I would expect 100 calls to mySetup(), but only 10 are occurring:
/**
* @BeforeMethods
* @Revs(10)
* @Iterations(10)
*/
class MyBench
{
public function mySetup(): void
{
reset_all_the_things_here();
}
public function benchSomething(): void
{
// irrelevant
}
}
What's the best way forward here? Is a change in @BeforeMethods viable? It would change the benchmark results for bench suites I've worked on so far, massively, but it would lead to more honest results.
I was working on laminas/laminas-servicemanager#93 today, and noticed that we still have the codebase sprinkled with blocks referencing #304
In fact, #304 has been closed as "old", but still applies today.
My assumption was that
@BeforeMethodswould be called before each call to said bench method: doesn't seem to be the case, and that leads to benchmarking warmed up caches too, which is a problem (especially if we're benchmarking said warmup).In following example, I would expect 100 calls to
mySetup(), but only10are occurring:What's the best way forward here? Is a change in
@BeforeMethodsviable? It would change the benchmark results for bench suites I've worked on so far, massively, but it would lead to more honest results.