Benchmarking Swift Code Properly with Attabench

Benchmarking Swift Code Properly with Attabench

Hmmmm, how fast is this piece of code? Let's find out!

When it comes to benchmarking the speed of code, it's common for people to boot a playground, throw in a Date object and calculate the time difference after a piece of code runs. That may give you a rough estimate, but it can also be very misleading. The difference between code running in debug builds versus release ones can be massive, and the size of the input (assuming we're benchmarking an algorithm) can also make a huge difference in the speed of a function. In this article, we'll see how to properly benchmark your Swift code. (Note that we're not talking about things like iOS startup time -- we're talking about the speed to run a specific piece of code.)

Introducing Attabench

Attabench is an open-source benchmarking tool created by Apple engineer Karoy Lorentey back in 2017 and has been my favorite benchmarking utility ever since I found it. The tool itself is a wrapper on top of Swift Package Manager that includes a benchmarking framework in where you can set up the code you'd like to measure, accompanied by a GUI app that lets you to see and customize how the result is presented:

Attabench

The measuring process is similar to how one would do with a playground, but Attabench's SPM abstraction not only compiles your app in the release configuration but also runs your code multiple times with different inputs, later providing a graph showing how your code behaves in function of the size of its input.

The project has technically been abandoned in favor of the Swift Collections Benchmark package that is roughly the same thing as Attabench, but the package is heavily command-line based which makes it a lot harder to use than Attabench. For that reason, I think that Attabench is still the superior benchmarking tool.

You can download Attabench's source code in its repo, but since the project has been abandoned a long time ago you may have issues compiling it with newer Xcode versions. My friend Rafael Machado was able to fix the compilation issues and provide a binary, which you can download here for simplicity.

Before you open Attabench, we must first create a benchmarking file. It doesn't seem like you can create this from the tool itself, so we'll do it by copying an existing one. The zip downloaded I linked earlier has a Sorting.attabench file -- copy it somewhere and open it with Attabench.

When you open the file, you'll see a benchmark that compares Swift's default sorting method with a custom Quicksort. If you click the Run button, Attabench will start continuously running the two methods with different input sizes and update the graph as it proceeds:

Alt

It takes a massive amount of time for Attabench to fully complete its run, but you don't need for all of it. I normally wait just a couple of seconds until the graph appears to be stable enough and stop it.

One interesting configuration you can play with is the input size in the top left of the app, which controls the minimum and maximum value that Attabench will use in its runs. In this case, the value between 1 and 1 million dictates the number of elements in the array that is going to be parsed.

Another important configuration is how is shown to you, available at the top right. By default the graph shows the average of how long it takes and on a logarithmic scale, but I'm not smart enough to understand what I'm supposed to do with that information. I personally like to disable everything and see precisely how long the code takes to run for each input size.

Now that you know how to use Attabench, let's see how you can create your own benchmarks. The .attabench file is a workspace that you can open as a folder, and inside you'll find a SPM Package.swift file. If you open it in Xcode, you'll be able to see and modify the benchmark:

let benchmark = Benchmark<[Int]>(title: "Sorting")
benchmark.descriptiveTitle = "Time spent sorting elements"
benchmark.descriptiveAmortizedTitle = "Average time spent sorting elements"

benchmark.addSimpleTask(title: "Swift.sort") { input in
    _ = input.sorted()
}

benchmark.addTimerTask(title: "Quicksort") { (input, timer) in
    var input = input
    timer.measure {
        input.quicksort()
    }
}

benchmark.start()

As you can see from the snippet, the benchmarking process is as simple as defining a benchmark object and registering the code that you want to measure.

The generic argument of the Benchmark represents the type of the input, which in this case is an array. You can modify it to be anything you want, as long you provide a function that generates the input for a given provided size. For example, if you'd like to create a benchmark where the input is a number, here's how we could define the Benchmark object:

let benchmark = Benchmark<Int>(title: "Calculate number of digits in a number") { size in
    return size
}

As you might've realized by now, size is a random number between the range you defined previously in Attabench.

To define a block of code to be measured, simply add a call to addSimpleTask (heh) with the measured content inside a closure:

benchmark.addSimpleTask(title: "Swift.sort") { input in
    _ = input.sorted()
}

By default, the entire content of the closure will be measured. If you'd like to have finer-grained control over what's being measured, you can use one of the lower-level abstractions like addTimerTask. In this example case, I use a timer task to make sure Quicksort's benchmarking includes only the time it takes to run the algorithm itself, and not the setup needed to run it (creating a mutable copy):

benchmark.addTimerTask(title: "Quicksort") { (input, timer) in
    var input = input
    timer.measure {
        input.quicksort()
    }
}

Finally, you can call benchmark.start() to run the benchmark. To update the Attabench app after changing the code, simply click the refresh button at the top-left corner.