Drupal benchmark results using AB and the simple things we did to get here

by , Web Developer

I was trolling around the Internet today looking for drupal hosting benchmarks and I actually had a little trouble finding something current. Dries has one comparing D6 on PHP4 vs D6 on PHP5 but that was clearly ages ago. I also realize that this is going to be out of style in about 12 minutes, will probably be fraught with contention, and generally mocked by everyone; but in the interest of those few souls out there that really actually just want to know what is reasonable to expect from production hardware under some load I want to post these stats anyways.

Our setup

We're using a gaggle of 1U SuperServers by Super Micro. The basic stats are here. We have them loaded with a pair of 4-core opterons and 32GB of RAM. Not entirely unaffordable nowadays for production kit.

These sit behind some firewalls and an F5 load balancer which helps make SSL a little quicker and makes sure that if things go down we can fail over to a different webhead. The truth is that all this redundancy stuff up front actually slows our pages down pretty heavily for small loads but when doing a lot of traffic on those happy spikey days it helps out a lot.

We've done a lot of the normal things to speed up the servers:

  • Turn on APC, this is huge!
  • Get rid of your .htaccess files and configure apache to do this at restart
  • Use Boost if you can't run Varnish, but run Varnish wherever you can
  • Turn off all the devel modules
  • Turn on as many of your drupal caching options as you can without breaking the site. Then figure out why the site broke and try to get those breaking caches online
  • Turn on your MySQL Query Cache, it's off by default and makes an amazing difference
  • Put MySQL on its own disk, put logs and even the MySQL binlogs on a different disk if you can
  • MAKE SURE YOU HAVE ENOUGH BANDWIDTH! Getting capped by your ISP will kill you during spikes since your servers will be resending lost data constantly

Commands Used

Since this article is about ab I'm not going to post our configs from SIEGE or JMeter.

Running from a box with an extremely fast network connection we were doing the following:

ab -n 10000 -c 100 -k -H 'Accept-Encoding: gzip,deflate' www.site-name.tld/

I like to run from a box within the network directly to the webhead as well to find out how much things are getting slowed down in the load balancers and firewalls. Since our webheads often do more than one site we have to use a Host header and the actual IP address as shown in the following snippet.

ab -n 10000 -c 100 -k -H 'Accept-Encoding: gzip,deflate' -H 'Host: www.site-name.tld' 192.168.0.27/

I also like to use relatively high concurrency and number of requests so that momentary spikes even out a little bit. There is a rule out on the net that you should take the average of three attempts for any benchmark and I think that's totally necessary.

We've noticed that our servers start to really crack around 500 concurrent anonymous users. Peak performance seems to be around 100-200 so I stick with that so that the differences in config changes are the most obvious. If I go from 100 #/s to 250 #/s I know it's a big change (or vice versa in a lot of cases).

Results

Without further ado, these are my results. I realize that there is certainly more that we should be doing to squeeze out performance and I also realize that my methods are pretty unscientific but I hope they give you an idea of what you might be looking for.

Internal - Test 1


Document Path:          /
Document Length:        10716 bytes

Concurrency Level:      100
Time taken for tests:   40.694054 seconds
Complete requests:      10000
Failed requests:        9993
   (Connect: 0, Length: 9993, Exceptions: 0)
Write errors:           0
Keep-Alive requests:    0
Total transferred:      112128752 bytes
HTML transferred:       106842680 bytes
Requests per second:    245.74 [#/sec] (mean)
Time per request:       406.941 [ms] (mean)
Time per request:       4.069 [ms] (mean, across all concurrent requests)
Transfer rate:          2690.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2  84.8      0    2999
Processing:    61  379 809.9    147   12255
Waiting:       53  344 799.4    134   12255
Total:         61  381 814.4    148   12259

Percentage of the requests served within a certain time (ms)
  50%    148
  66%    170
  75%    265
  80%    345
  90%    599
  95%   1371
  98%   3159
  99%   4725
 100%  12259 (longest request)

Internal - Test 2

Document Path:          /
Document Length:        10683 bytes

Concurrency Level:      100
Time taken for tests:   29.175060 seconds
Complete requests:      10000
Failed requests:        5389
   (Connect: 0, Length: 5389, Exceptions: 0)
Write errors:           0
Keep-Alive requests:    0
Total transferred:      111006011 bytes
HTML transferred:       105724371 bytes
Requests per second:    342.76 [#/sec] (mean)
Time per request:       291.751 [ms] (mean)
Time per request:       2.918 [ms] (mean, across all concurrent requests)
Transfer rate:          3715.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5 123.8      0    3033
Processing:    41  273 445.8    135   11841
Waiting:       36  245 430.6    124   11840
Total:         41  278 463.5    136   11851

Percentage of the requests served within a certain time (ms)
  50%    136
  66%    150
  75%    176
  80%    287
  90%    491
  95%   1305
  98%   1534
  99%   3118
 100%  11851 (longest request)

External - Test 1

Document Path:          /
Document Length:        10472 bytes

Concurrency Level:      100
Time taken for tests:   31.161653 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      110620000 bytes
HTML transferred:       104720000 bytes
Requests per second:    320.91 [#/sec] (mean)
Time per request:       311.617 [ms] (mean)
Time per request:       3.116 [ms] (mean, across all concurrent requests)
Transfer rate:          3466.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13 199.5      0    3051
Processing:    46  273 414.5    138    4558
Waiting:       41  246 404.3    126    4545
Total:         46  287 457.2    139    4558

Percentage of the requests served within a certain time (ms)
  50%    139
  66%    156
  75%    299
  80%    336
  90%    482
  95%   1119
  98%   1580
  99%   3143
 100%   4558 (longest request)

External - Test 2

Document Path:          /
Document Length:        10472 bytes

Concurrency Level:      100
Time taken for tests:   29.336590 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      110000000 bytes
HTML transferred:       104720000 bytes
Requests per second:    340.87 [#/sec] (mean)
Time per request:       293.366 [ms] (mean)
Time per request:       2.934 [ms] (mean, across all concurrent requests)
Transfer rate:          3661.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  95.1      0    3034
Processing:    50  264 447.7    129    4532
Waiting:       45  240 435.7    119    4528
Total:         50  268 459.6    129    6128

Percentage of the requests served within a certain time (ms)
  50%    129
  66%    143
  75%    162
  80%    295
  90%    423
  95%   1307
  98%   1532
  99%   3125
 100%   6128 (longest request)

Comments

Posted by greggles on . [Reply]

The standard deviation numbers are bigger than the mean/median?

That seems odd, especially for a test with so many requests.

Are these boxes isolated from other tasks or are they doing something real at the same time that you're running the tests?

Posted by peach - sooperthemes on . [Reply]

I don't think this test looks like a joke, you got some amazing hardware and a good test.
I once this a little test on my windows vista laptop and it is referenced wikipedia as the ultimate evidence that Drupal is fast...

Are you using boost? Because then your requests per second are looking a little low. What is the bottleneck?

Posted by dalin on . [Reply]

"Get rid of your .htaccess files and configure apache to do this at restart"

And by "get rid of" I assume you are just doing something like this in your vhost config:

Include /path/to/site/.htaccess
AllowOverride None

Which doesn't help too much for the main Drupal .htaccess file, but helps a lot for the one in the files directory. And doing so means that you need to restart Apache whenever the file is changed. So if I choose to do this step I usually only do so for the files directory since its .htaccess file never changes.

"Turn on your MySQL Query Cache, it's off by default and makes an amazing difference"

Or better yet, run mysqltuner2 or dbtuner on your server.
https://launchpad.net/mysqltuner
http://drupal.org/project/dbtuner

Turn the query cache on, increase it from the default, but don't make it any bigger than about 128-256MB or you'll see a serious slowdown.

Posted by dalin on . [Reply]

When I saved my last comment, this error came up:

* The selected file /tmp/fileBd74kr could not be uploaded, because the destination sites/default/files/xmlsitemap/sitemap.xml.gz is not properly configured.
* Unable to load site map. Make sure that there is an xmlsitemap directory in your files directory and that it is writable by Drupal.

Posted by Khalid on . [Reply]

Solid advice.

The list of points you mention is good solid advice.

A few comments on them though:

  • For .htaccess, we have not seen much evidence that rolling it into the vhost would have a measurable response. At least when you are using the .htaccess that Drupal ships with and not much more. And if your directory structure is shallow (e.g. /home/site.example.com/www/.htaccess), not much searching in the directories above will occur. Maybe you have directories too deep, or more complex rules?
  • More important than Boost/Varnish is to have memcache and page cache turned on. In lieu of Varnish (and even nginx), you can have Apache as a MPM Worker threaded server, and PHP as a FastCGI. This solves the memory footprint for Apache prefork, and makes serving static content very fast.
  • The devel module can stay, even on a live high traffic site. We say that from experience. The devel data is visible only to an admin user, and hence regular users or anonymous users will not see anything. You can disable the collection of query data in memory to save some RAM, but that is about it.
    • vosechu's picture

    Posted by Chuck Vose on . [Reply]

    Thanks for all the awesome responses, I really appreciate the time you took to read and help me. This is why I've fallen in love with the Drupal community.

    @greggles - That's a great point, we're actually serving normal pages at the same time. I tested at a somewhat off-peak time but one of our clients is doing decent traffic all the time so it seems too expensive to test the whole stack without traffic. Maybe when we have a scheduled downtime I can try it out though. :) Thanks for the note about the statistics aspect though, I hadn't thought about much more than the req/s.

    In the past we've had some trouble with the network being a little lossy which seems like it could cause that variability but since we upgraded it's gotten a lot better. It could still just be the network though, not sure.

    @peach - Thanks for the comment, I was worried I would just look like a young man touting his new equipment but I'm glad that it didn't come across that way. We actually aren't running Boost on these particular sites but I have seen some wonderful results with it on another of our client's site. They're behind a CDN now so boost hasn't been a the waterfall that it was in the beginning so I don't have great numbers for that. For this particular client we ran into some serious difficulties with Boost but because of the site it's extreeeemly difficult to make big changes like this; it has some 60 sites in the multi-site setup and it seems like any significant change breaks at least three of them. :P

    @dalin - Awesome advice on MySQL, I hadn't run across mysqltuner2. I think I used a different tool that did a similar thing but I forgot to mention it in the article (actually, I got interrupted by a meeting and had to finish quickly :P). And you're totally right, I should be more explicit about turning off .htaccess; I'd hate to have someone blindly follow my advice or even think that I was advocating that the default Drupal version was in some way a mistake.

    Also dalin, thanks for the note about our xmlsitemap config, I'll see what I can do about that on monday.

    @Khalid - For .htaccess the results were fairly significant for us for a number of reasons but I really didn't go into them enough at all in the blog, my apologies. For us the reasoning was twofold: a central place to put things like Boost configs is totally killer in our environment. It's not perfect of course, but being able to include a common.conf makes the config a little more sane. The second reason was that the apache performance page mentions that having Allow Override set to anything other than None causes extra lstats for every single request for every folder in the tree up to the point where the file is. (http://httpd.apache.org/docs/2.0/misc/perf-tuning.html) It's not a big deal, but it's helped our read i/o a bit which is something we were really needing. I guess there's a third reason which is that we're experimenting with a VirtualDocumentRoot setup which is easier using .conf files but that's really not about tuning and I have no idea the performance impacts of this change. :)

    I think that I must have not set up my memcached servers right or they don't help with ab at all. Have you noticed significant changes with memcached in your testing? We had another client that asked for memcached but I don't know if the query cache on MySQL was just performing well enough on the db server that maybe I didn't notice. We also had the MySQL server on some extra nice kit while the memcached servers were on the same server as apache so that could have affected my results. We used the following recipe for our config pretty much exactly, do you have any additional advice or thoughts about why we didn't see much change?(https://wiki.fourkitchens.com/display/PF/Using+memcached+with+Drupal+or+...)

    Thanks also for the note about devel, I'll have to revise our deployment script to leave those on. I think they're terrifically helpful but I guess I just assumed that they would negatively impact performance on a major scale. I'm sure that the theme developer module is pretty rough, but I think you must be right about things like dpm being almost no impact for non-admin users.

    Add new comment