Thursday, November 23, 2006

Where'd the bandwidth go?

Last month I celebrated my 2nd anniversary here at the Foundation -- easy to remember, as it falls on the same day as Mike's birthday. After my very first day on the job, I was invited to celebrate with the then very small Foundation team, where we enjoyed finger foods and beverages. I also got to meet a few Eclipse celebrities, such as John Duimovich and Mr. SWT (although they don't know who I am -- we sysadmins work in the shadows.)

Anyway, in these last two years, I invested a significant amount of time optimizing our bandwidth usage. When I started, our then 50 Mbps Internet connection was always 100% saturated -- not only because Eclipse was popular, but also because picking a mirror was a lot of work. With some redesigned downloads pages, our bandwidth dropped to 30 Mbps, but eventually, over several months, it crept back up to a constant 50 as more projects were added, and as more people rushed in to get their Eclipse wares. We eventually caved in and added 10 Mbps because it was the right thing to do.

Well, we eventually filled that up too, so more optimizations were in order. We learned that mirroring entire nightly builds was costly, compared to hosting the relatively few downloads ourselves, so out they came. We also created, with the same idea as nightlies, in that "older" builds don't get many downloads, so why mirror them? Wow, down to 47.

Well, here we are, November 2006, and we're back at square 1 -- since the end of September we've been hovering around saturation, at 60 Mbps. Except now, unlike back then, there's not much left to optimize -- we're using our mirrors very efficiently, and other than cutting out bits and bytes from web pages here and there, I'm running out of ideas.

I think the moral of the story is that optimizing is pointless Eclipse has a vibrant community, and its rapid popularity and momentum gains seen throughout 2005 and 2006 haven't ceased or even slowed down -- and that we sysadmins are trying desperately to keep up with the demand!


Post a Comment

<< Home