How to migrate from BenchmarkTools to Chairmarks

Chairmarks has a similar samples/evals model to BenchmarkTools. It preserves the keyword arguments samples, evals, and seconds. Unlike BenchmarkTools, the seconds argument is honored even as it drops down to the order of 30μs (@b @b hash(rand()) seconds=.00003). While accuracy does decay as the total number of evaluations and samples decreases, it remains quite reasonable (e.g. I see a noise of about 30% when benchmarking @b hash(rand()) seconds=.00003). This makes it much more reasonable to perform meta-analysis such as computing the time it takes to hash a thousand different lengthed arrays with [@b hash(rand(n)) seconds=.001 for n in 1:1000].

Both BenchmarkTools and Chairmarks use an evaluation model structured like this:

init()
samples = []
for _ in 1:samples
    setup()
    t0 = time()
    for _ in 1:evals
        f()
    end
    t1 = time()
    push!(samples, t1 - t0)
    teardown()
end
return samples

In BenchmarkTools, you specify f and setup with the invocation @benchmark f setup=(setup). In Chairmarks, you specify f and setup with the invocation @be setup f. In BenchmarkTools, setup and f communicate via shared local variables in code generated by BenchmarkTools. In Chairmarks, the function f is passed the return value of the function setup as an argument. Chairmarks also lets you specify teardown, which is not possible with BenchmarkTools, and an init which can be emulated with interpolation using BenchmarkTools.

Here are some examples of corresponding invocations in BenchmarkTools and Chairmarks:

For automated regression tests, RegressionTests.jl is a work in progress replacement for the BenchmarkGroup and @benchmarkable system. Because Chairmarks is efficiently and stably autotuned and RegressionTests.jl is inherently robust to noise, there is no need for parameter caching.

Toplevel API

Chairmarks always returns the benchmark result, while BenchmarkTools mirrors the more diverse base API.

Chairmarks may provide @belapsed, @btime, @ballocated, and @ballocations in the future.

Fields

Benchmark results have the following fields:

Note that more fields may be added as more information becomes available.

Nonconstant globals and interpolation

Like BenchmarkTools, benchmarks that include access to nonconstant globals will receive a performance overhead for that access.

However, Chairmarks's arguments are functions evaluated in the scope of the macro call, not quoted expressions evaled at global scope. This makes nonconstant global access much less of an issue in Chairmarks than BenchmarkTools which, in turn, eliminates much of the need to interpolate variables. Consequently, interpolation is not directly supported. You can always throw an @eval in front and then use interpolation if you would like, though this will leak memory.^[1]

Four possible ways to avoid nonconstant globals are to put the @b call in a function, put the global access in the setup or initialization phase, make the global constant, or use @eval and interpolate it. For example,

julia> x = 6 # nonconstant global
6

julia> @b rand(x) # slow
38.449 ns (2.01 allocs: 112.449 bytes)

julia> f(len) = @b rand(len) # put the `@b` call in a function (highest performance for repeated benchmarks)
f (generic function with 1 method)

julia> f(x)
15.318 ns (2 allocs: 112 bytes)

julia> @b x rand # put the access in the setup phase (most concise in simple cases)
15.507 ns (2 allocs: 112 bytes)

julia> const X = x # make the global constant
6

julia> @b rand(X)
15.448 ns (2 allocs: 112 bytes)

julia> @eval @b rand($x) # use eval and interpolation (most familiar, but leaks a small amount of memory)
15.620 ns (2 allocs: 112 bytes)