Friday, March 13, 2009

Server performance -- solving disk bottlenecks

In February we added lots of disk space to the server farm.  There were two reasons for this:

  • We were running out.

  • Our current disk arrays were doing too much, hurting disk performance.

How do you know if your disk systems are too busy?

  • look at the activity lights on the drives.  Two of our 7-disk array's lights were mostly on all the time.

  • look for excessive IO Wait time using top.  Clear indications are high wa% as well as processes in the D state.

Disk bottlenecks at occur because so many files need to be read and updated at any given time that the disk heads are constantly moving (seeking).

How do you solve a disk seek bottleneck like this?

  • Avoid going to disk altogether by adding RAM [1]. Unused RAM is used for write buffers and file cache.  Adding 8, 16 and even 32 GB of RAM can make a sluggish system run like new.

  • Add disk spindles [2].  One disk can only seek so many files at a time, so the more disks in your array, the faster the seek times.

Our budget was quite tight, and I needed to solve both the low disk space and high disk seek problems with minimal money. I opted for a 16-disk array of 750G SATA drives, but no additional RAM.  Although SATA drives are typically slower than their high-end SCSI counterparts, they are much, much cheaper, allowing you to get lots of disk space and awesome seek performance from a higher number of drives.

Since implementing the new array, disk wait time went from a whopping 50%  to 12% on our file server, and from 40% to 2% on the build server.  Money well spent!

[1] If performance is critical, consider using a ramdrive for frequently accessed files.  Beware of writing there, though.

[2] Solid-state drives will definitely help out here, as soon as they become more accepted and available in entreprise-class hardware and environments.


Post a Comment

<< Home