AutoBenching Capacity

When you are building a service it's important to know early on what kind of performance and scalability you can expect to achieve. However. measuring this isn't an easy task and there are lots of variables to take into account - the hardware, the bandwidth, the disparity between production and development platforms, logging while in development, etc.

I've been working on a new project recently using a strict platform model -- interaction between the various pieces is all via restful http services only (so nothing except the api has access to the underlying datastores etc). This means that we should be able to scale out horizontally very easily by throwing some load balancers between the various layers and that we should be able to test each of the interaction layers with a http load testing tool to get some idea of our capacity and performance under load.



There are a variety of different products out there, from incredibly expensive managed test suites like LoadRunner from HP to free product suites such as Apache JMeter. However, both are way more complicated than I was looking for right now, so I thought I'd try out Autobench.

It was a bit of a pain to get Autobench installed initially on our Centos vm's. This is not Autobench's fault -- Centos comes with a very restrictive set of available packages with the default configuration. Autobench depends upon httperf (also from HP), which isn't in this set of packages. It is luckily in the extended packages from rpmforge. After installing httperf I managed to build Autobench cleanly after git cloning it's repo.

Autobench is a command line tool which basically calls httperf repeatedly, increasing the number of connections opened per second each time. The command line interface is a little obscure but it comes with a pretty nice man page and Autobench writes a tab separated results file containing the metrics gathered via httperf.

Even better, it includes a tool called bench2graph which runs gnuplot on the output file. So, after playing around with it for a while I generated this graph. I've inverted the chart background to show the trend lines better.:

Request Rate / Response Time - No Index

 X axis is the number of connections opened a second. The two series (Response Time and Request Rate) both map onto the Y axis. Response time is plotted against milliseconds and request rate is in requests per second.

 What we have here is the stats gathered from load testing our new browse API, which is a thin python layer with a MongoDB back end and running in the Flask framework. What our graph tells us is that we can only service 30 connections per second before our response time starts to suffer. This isn't a great statistic! Now, we haven't done any performance enhancement work so lets see what happens if we add a valid index to our MongoDB store.

We can see immediately that the situation is much improved! We have a nice trend line for the request rate with a saturation point of roughly 110 simultaneous connections per second and our response time is lovely and flat before that saturation point. even better, we can see that the service doesn't tank horribly after saturation.

With a tool like this, it will be easy to see what difference changes we make will have on our final capacity per unit. This means that as we add caching or other index improvements we can prove that they make a material difference. And because it's a command line tool we can easily automate this process to run performance tests on a code push.

It's manual and a little hard to get used to but I was really impressed with Autobench.