Tuesday, October 18, 2005

A year at Eclipse

Eighteen days ago I celebrated one year as webmaster@eclipse.org. What better way to reflect on my experience here at Eclipse than by looking at our bandwidth usage over the last year:

(There's a picture here. If you don't see it, click my post)

1. Moving the download servers from the IBM facility to Magma. Within seconds of plugging them into the switch, the disks go mad and lights flash all over. With a bewildered look, the Magma technician asks, "Is that normal?" Yup. Within a few hours our 50 Mbps cap was hit.

2. Merry Christmas. The holiday season was awful quiet.

3. Happy New Year! Saturated once more. Looking into it, we determined that the cause of our constant saturation was that our downloads page makes it hard to use mirror sites. Most people just scroll down and pick "Main Download Site"! Nostalgia: October 2004 downloads page.

4. Our bandwidth usage is now much more efficient as a result of reworking the way downloads are handled - we instantly dropped to about 30 Mbps as people are now using mirrors instead of the main download.eclipse.org server. We even considered reducing our cap from 50 to 40, or even 30, but we didn't, and lucky us - items 6 and 7 came along to gobble up the slack.

5. Transition starts to new hardware. The previous set of independent servers are being replaced with a powerful cluster of servers. Thanks again to HP, IBM, Intel and Novell for everything!

6. EclipseCON's success brings tons of visitors to Eclipse.org. Good thing we had freed up some bandwidth. Our site never "recovers" from the good buzz at EclipseCON, as our bandwidth stays at a higher level for several weeks. I like that.

7. Eclipse 3.0.2 is released, which brings us back to saturation. Over-saturation. A few of weeks of pain make us realize we need to add bandwidth temporarily, which we do. After the release rush is over, we bring our bandwidth cap back to its normal setting.

8. We're back to full-time saturation, despite using our mirrors more efficiently. This type of saturation is good - our popularity has increased dramatically in the last several months!

9. Eclipse 3.1 is released, but we were ready - we lifted our bandwidth cap altogether. Bring it on, we can handle it!! Wrong. After a few hours, our website came to a crawl. You guys wanted 3.1 like a kid wants candy! Mirrors couldn't even connect to fetch the new release. I didn't sleep for days. I lost 60 pounds. People blasted me for not putting up torrents. I still suffer posttraumatic stress disorders to this day when Kim mentions the word "release".

We reinstated our 50 Mb bandwidth cap 5 weeks after release; however, Eclipse had become so popular that we increased our permanent cap to 60 Mbps shortly after.

A. Eclipse 3.1.1 is released. We learned a lot from 3.1, and everything went smoothly.

B. Language packs for 3.1.1 and for many projects are made available, thanks to IBM. Popular items! They make a substantial dent in our bandwidth, but not enough to bump it up.

It was a great year. Eclipse gained so much popularity, what a vibrant community to be part of! Kudos to the developers, Eclipse and its plugins are a fine piece of work!

Friday, October 07, 2005

Mirror, mirror on the ... list

I got a message from one of our mirrors this morning:
I'm disabling I-* and R-* [builds]. September was the 2nd month at over 7TB transfer, which is significantly over my transfer limit. I'll continue to mirror S-* [builds].

Was fun while it lasted... certainly the biggest webalizer numbers I've ever seen.

Seven Terabytes. 7,168 gigabytes. Seven million megabytes. That's 1500 DVDs, full to the brim. And this is only one mirror out of 35.

Eclipse is gaining in popularity, and fast. Note that this particular mirror isn't even an Update Manager mirror - meaning they only provide the zip files. Eclipse.org wasn't saturated until the Update JARs made it to the update site on tuesday morning. Since then, our bandwidth has been pretty much maxed out, despite adding an extra 10 megabits to our permanent link AND despite waiting for Update mirrors to fetch the Update JARs before making them publicly available.

As a side note, I've observed that even in a tech-savvy community like Eclipse, the vast majority of downloaders keep fetching the SDK through plain old http as opposed to using the Bittorrent links. iBiblio reports about 1332 completed downloads for the Win32 SDK, whereas our download stats amount to 145,558 http downloads for the same file.

Wednesday, October 05, 2005

Online help and docs

Eclipse.org currently runs an Infocenter help at http://help.eclipse.org/help31/ but it's only for the Platform projects (SDK, PDE, etc). Because it's running Eclipse, I have the ability of dropping in documentation plugins from other eclipse.org projects, such as Birt (which I added to the above Infocenter, just for fun).

Should all eclipse.org projects jump in to host their latest release documentation on the eclipse.org Infocenter? As discussed in bug 100635, the index pane could become quite cluttered, and search could return many unrelated results. What do you think? Post comments to the bug only, not here.

Oh, and before you ask, running a separate Eclipse instance for each doc is way out of the question.