Tuesday, April 20, 2010

Happiness is keeping it in RAM

Last week I deployed a new download.eclipse.org mega-machine.  If you haven't noticed, that's great.  The transition was completely transparent.

Reduced disk seek time on our file servers by maximizing file cache hits is what this is all about.  Here's a bandwidth graph of what our download server is sending to the Internet -- over 70 Megabits/sec (the green line) during the Eastern timezone daytime.

In return, this is what the download server is fetching from the NFS file server (the blue line).  You can see that as the server came online last week, the file cache was empty, and all the download requests needed to be fetched from NFS.  But today, NFS is serving less than 5 Megabits/sec to the download server.

70 Megabits out, 5 Megabits in.  Good cache hits there.

But in practice, since our bandwidth is limited, you won't see a faster download speed.  But your download should start instantaneously.  However, the biggest gain is achieved for services that need to use the NFS file server, such as CVS, SVN and Git.

But this is only the beginning ... wait until you see what we're doing with the MySQL database servers...

Monday, April 12, 2010

Setting up a download server.. How much RAM do I need?

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

If you've been following along, you know that I received a bunch of hardware to upgrade Eclipse.org.   'Keep it in RAM' is my thinking with all this.  Currently, our servers spend a lot of time waiting for disk resources, so the benefit of keeping requests in RAM is twofold: cached files are served quickly, and disk resources are freed for faster access to disk.

Right now I'm in the process of setting up a high-capacity download server to replace download.eclipse.org, so the question is -- how much RAM do I need?

I started by examining the combined Apache logs of download.eclipse.org for any given day.
zegrep "GET .* 200 " download.eclipse.org/access_log.3.gz | awk '{print $7}' > filelist
filelist contains 2,439,051 successful "GET" requests

sort filelist | uniq -c | sort -nr > filelist-sorted
filelist-sorted contains 73,184 entries

So each day our server only reads about 75,000 files -- but serves them 2.4 million times.  There is a huge potential for cache hits right there.  With a small perl script, I gathered the size (on disk) of those 73,184 files.  Total: 25G.


open(FILE, "<filelist-sorted");
while(<FILE>) {
        $_ =~ s/\n//;
        ($c, $f) = split(/ /, $_);
        $size = -s "downloads" . $f;
        print "$size $c $f\n";

So if I get 24G of RAM, I'm sure that most of the file requests will come from cache.  Actually, if I load up the numbers in a spreadsheet, I see a wicked long-tail distribution.  In fact, 134 files are fetched at least once each minute and account for 43% of all requests.  If you consider the RAM requirements of the OS and Apache, 24G would be great -- for today's needs.  What about next year?

Considering it's cheaper to buy RAM when it's popular (commodity), I put 64G of RAM in the new download.eclipse.org.  It should be more than sufficient to hold the entire week's worth of download files, file attributes and such while keeping disk requests to a minimum.  It will also have plenty of RAM for the OS and Apache, even when things heat up in June.

We're also moving to the Apache worker MPM for download.eclipse.org.  It is multi-threaded, so with high client counts, it consumes much less memory than the prefork model.  PHP files (which are only a very small fraction of the hits) will be served over FastCGI.

So there you have it.  Your downloads won't necessarily be faster since we are limited by bandwidth, but they should begin faster.  The new setup will also free disk resources for those files that cannot always be cached, such as CVS, SVN and Git.  Win-win!

Thursday, April 01, 2010

Git vs. IP provenance -- DVCS with a twist

With a Distributed Version Control System (DVCS) such as Git, there is no reliance on a central system.  This allows developers to easily pound out and exchange code amongst themselves.

This also causes an Intellectual Property (IP) nightmare for OSS foundations like Eclipse, where IP and IP provenance are keystones to the organization.

For a DVCS to work at Eclipse.org, we need to make some compromises. The Eclipse Foundation needs to know where the code comes from, and who wrote it. Picture this:

In the case above, Jimmy and Jina are not known to Eclipse.org, but they are working together to help Billy, an Eclipse committer, by contributing code.  Since Git does not maintain a "push" log, if Billy's push to git.eclipse.org was to succeed, we would have a git commit history that would contain entries of people that we do not know.  In fact, since the email address is configured by the user, Eclipse has no real way of determining the validity of the information.

In the above case, the push will fail.  So how does Billy make this work?

The correct workflow is seen above.  Jimmy and Jina implement a feature and wish to have it included in Eclipse.  They submit a patch to Bugzilla, where it is then reviewed by a committer by following the Eclipse Development Process and Due Diligence guidelines.  From here, the committer will submit the code contribution to IPZilla if need be.

If the code passes all the required reviews, Billy incorporates it into his local Git clone and pushes the code to Eclipse's Git server.  Since the Git commit log contains an entry for the Author name/email, Committer name/email and a commit comment, the names of the original authors, as well as the bug number can all be referenced for easy traceability.

This process does remove a bit of the "D" in DVCS, but if we follow a few simple rules, we can still take advantage of the benefits of Git while maintaining a clean IP trail.