Friday, June 29, 2007

Europa in full swing

The Europa release went off this morning without any technical problems, other than it took longer than expected to copy files to our own servers. The actual "we're ready" was declared at around 9:45am ET.

Our servers kept up the pace: I saw a maximum throughput of 249 megabits/second, sustaining over 1,500 clients. Within a few hours we had registered thousands of downloads of the SDK and the various Eclipse packages.

Things are getting a bit quiet as the weekend rolls in for North America, but next week will be our busiest.

Eclipse Europa is here!

Tuesday, June 26, 2007

A cool downloads page - just in time for Europa

The Phoenix team (well, mostly Nathan) has been working on a new "main" downloads page prototype to replace the existing one. The purpose here is to a) highlight the user-oriented Eclipse packages created by the EPP and b) to highlight some of the Eclipse distros that commercial vendors are offering the community.

I think the page looks great. There's a good mix of information, useful links and visual aids without overcrowding the page -- something I valued dearly when I originally redesigned the main downloads page in early 2005. With this new page, the newcomer should (finally) have a better idea what to start with, whereas the veteran Eclipse hacker should already know where to find the hot-off-the-build-machine nightly build for today's testing.

Let's focus a bit on a) and b) if you will.

a) represents a cornerstone for our website - I mean, this is absolutely HUGE. HUGE! Users have been insulting asking us for user-oriented (and solutions-oriented) packages for YEARS, and thanks to our friends on the Packaging project, this is becoming a reality. There is still no native installer (yet) so we still need to contend with unzipping Eclipse, but just having an all-in-one package for the C developer, or the JSP developer, is notable progress. Someone just needed to step up to the plate and do it.

b) is also an interesting concept. Here we have commercial entities, companies that exist to turn a profit, who are essentially packaging Eclipse and offering free downloads. Of course, they will include links to their commercial offerings and trial versions, but the overhead is very lightweight: no upselling, no required registration, no pesky advertisements and no opt-outs. They're spreading the word about their commercial products by playing nice and being useful. Coolness all around.

Vservers can improve you life (if not your diet)

Denis, Matt, and I have been talking about lots of different architectures we could use to improve some of the infrastructure at We have some miscellaneous machines which aren't part of the core infrastructure but do some important jobs. And then we have a couple of host servers that hold vservers for some of the projects, which are mostly full and which are hosted on local disks.

We don't have a lot of rack space, or room for lots of physical servers but we keep finding that we want separate servers for various things, and that we have whole servers sometimes dedicated to things which don't really need to occupy an entire server. How do you re-allocate all of that without making a huge mess? Enter vservers!

In reading through lots of high availability documentation while working on our NFS problems awhile ago, Denis and I both read about a nice idea Novell used in one of their demonstrations. They had several vserver images hosted on a share and used multiple vserver host machines to mount the share. Their idea was to have the vserver reloaded on another server in the event that the original host server crashed, thus providing some high-availability for the vserver. For the non-critical servers we're talking about we don't need high availability. But if we use that architecture of interchangeable vservers and images hosted on a share, we can move vservers around between hosts on the same subnet however we like to best suit resources without affecting a thing about how they run. This is not a new idea, but it sure is a good one. I hadn't been thinking about it in this context, but talking to Denis today we struck on that as a good strategy. We haven't made any firm plans yet about which servers will move, but you can bet we'll be employing more vservers in the infrastructure soon.

Monday, June 25, 2007

Ready for Europa

This morning I de-listed all our mirror sites so I can test our shiny new bandwidth capacity. It's not permanent, though -- the additional 230 megabits are to support the large demand for Europa. By the end of August, all that new bandwidth will be taken (pried) away from me.

The test was successful, as we managed to saturate all the new bandwidth, and we were able to sustain all our downloads without any mirrors -- for about an hour and a half. At one point, our IBM megaserver was pumping out 162 megabits all by itself, serving well over 2100 clients without any effort at all. We maxed out our entire 300 megabits within an hour of delisting all the mirror sites.

After about 30 minutes of hitting the saturation point, reaching a peak of 3600 http clients, I re-enabled our mirror sites once again. Our testing was successful, our servers and switching gear was still alive and well, and we were serving all we possibly could with 300 megabits, so to keep up this insanity was simply pointless.

Conclusion: we're ready for the Europa onslaught, but only if our mirrors are present and ready to help (as they always are). There is simply no way we can handle Europa on our own with only 300 megabits.

Friday, June 22, 2007

Mirror site appreciation day - June 25, 2007 !

I'm declaring Monday, June 25 as "Mirror Site Appreciation" day.

I just got a call from our ISP saying our extra Gigabit connection has just been plugged in, and I need to test it to be ready for the Europa release on June 29. Actually, I don't need to test the connection itself -- I need to test our servers, switches (and our ISP's routers). One thing learned from last year's Callisto is that any server can saturate a Gigabit connection when only a handful of clients are accessing it. Things get infinitely more complicated when thousands of clients pour in, requesting a whole bunch of files.

So, how do we test this? Easy! We disable all our mirror sites and handle *all* of our download traffic. On Monday, when you go to to download something, use your best Matrix voice and remember that "there is no mirror".

As was the case last year, I reserve the right to chicken-out of this insanity at any time during the day if I feel we can't handle it. Happy downloading!

Thursday, June 21, 2007

My hard drive has code?

Have you ever needed to update the code that makes your hard drive work? I had never even stopped to think that hard drives actually have software that makes them work, until I needed to update the code on two of our hard drives.

It all started in March 2006, when our central file server just seemed to stop doing what it's supposed to, without any error log or anything, forcing us to fail over to our standby server. This "crash" would repeat itself several times in March and April '06, and about a dozen times throughout the end of 2006 and into 2007.

We eventually blamed a host of things, including the Linux quota daemon, NFS and bad RAM, until we realized that our disk array would, for no known reason, become overwhelmed with disk writes until it simply stopped. We didn't know why, so we eventually wrote a bunch of scripts that would try to identify an upcoming crash and simply halt network traffic to give the server a "break". This was an effective workaround which led to better webmaster sleep at night, but the problem was still there, and we feared that the Europa release would cause the server to crash of exhaustion with all the added demand.

Long story short: two of the drives in our 14-disk tray were a different model, and there was a code update for them that fixed a problem where the disks "Entered read/write protection mode when a self-test timeout occurs". The update sounded interesting, so I went ahead and applied it to the two disks. Updating disks in a live (and busy) array is scary stuff, but the IBM tools make it easy and painless.

It's still too early to tell if this fixes the crashes we've been having, but I'm confident it does. Talk about weird bugs.

Tuesday, June 19, 2007

New toys for committers

I haven't been giving our beloved Committers as many toys as in the past (new sites, new apps, new tools, etc.), and I feel bad. Last year I'd give out new toys every other month!

Well, here's a new toy for them: Blogs. In response to bug 179857, I set up a wpmu site on the Eclipse servers to host Committer and project blogs. It took me a while to hack this into something we can use, as it had to authenticate against the Bugzilla database, it had to differentiate committers from non-committers and it had to look like the rest of our site. The wpmu code isn't what I consider great, so that only added to the complexity of playing around in it.

Next month (after Europa is released and things calm down a bit) we'll also be implementing new versions of old toys, such as Bugzilla 3.0. It's a big upgrade, and I'm sure the committers will love that one.

Oh, and beer. I'll be giving them beer. And pizza. That counts for a lot, doesn't it?

Spamming Bugzilla

This morning, a computer in Russia was used to post SPAM attachments to our Bugzilla system. The fun part is that, because Bugzilla doesn't allow anonymous posting, one must go through the trouble of registering for a Bugzilla account using a valid e-mail address, check their e-mail for the Bugzilla confirmation, log in, and then attach their SPAM. Certainly there are easier targets, and this likely explains why we don't get too much SPAM on Bugzilla.

Within a few minutes of the first attachment, three committers sent an e-mail to the webmaster box. Upon seeing the subject line in my e-mail popup, I dropped everything to quickly disable the Bugzilla account, block the Russian subnet on our firewall, then remove the offending attachments and comments.

So this loser spammer just spent a good chunk of time doing all this, and in the end, for what? To advertise which porn web sites? See for yourself.

Loser. Go away.

Monday, June 18, 2007

Europa committer celebrations

Europa is only 11 days away, and now is the time to start planning the celebrations. Last year, I invited committers in the Ottawa area to join me at the Foundation's offices for some snacks and, uh... "orange juice" to celebrate the Callisto release. Kim suggests we repeat the celebrations this year (the last line of this document is quite subtle) and I'm sure I can convince Mike to allow me to spend a few dollars on Pizza and beverages.

If we have a get-together at the Foundation office again this year, I promise to make the release more exciting than just watching me type a command and commit a file, but I'm open for suggestions: what do the Ottawa committers want to do to celebrate?

Friday, June 15, 2007

More bandwidth for Europa

I just placed the order for more bandwidth to handle the Europa demand -- not a moment too soon, as the many builds and testing phases happening right now are putting a strain on bandwidth. In addition to our regular 100Mb connection, we're going Gigabit for the additional bandwidth. But before you get all excited and think that will be blazing fast, we'll be limiting the Gigabit connection to 200Mb to keep costs reasonable. The fun part here is that we can simply increase the 200Mb limit as we see fit, if demand exceeds supply.

Like last year's Callisto, we'll be providing our mirror sites with dedicated access to the files before they are generally available. This is for two reasons: 1. so that our site doesn't crumble from the pressure of being alone with Europa, and 2. so that the first mirrors to report having the Europa files don't have to carry all the burden of being the only mirror sites out there. Pre-loading the mirrors just makes sense.

Europa is coming -- 15 days to go!

Wednesday, June 13, 2007

Webmaster blog gets a real author

When Matt and Karl joined the ranks as webmaster, I invited both of them to be authors on my blog. After all, it's not my blog, it's the webmaster's blog, and we all wear the same shoes. They kindly declined, most likely because they were too shy at the time. However, Karl told me 'yeah, I have stuff to say' and accepted my second invitation (no word from Matt yet). So, without further ado, I welcome Karl to my blog.

Looks like I'll need to update my feed title and face on PlanetEclipse now.

Conferencing is coming...

Many Eclipse community people are probably familiar with the Foundation's conference calling solution, having attended meetings in some manner via the dial-ins. Well, not surprisingly, that has turned out to be a relatively expensive service and the cost has prevented us from being able to roll it out more widely to the community. It's something people seem to want, and we'd like to provide it--while getting off of the more expensive service we're using now. So we talked about it a lot at the Foundation and Bjorn entered this bug to get a reference point for the desired replacement. Denis (my illustrious boss) has mentioned this on many different occasions, but we're actually getting close to putting it out there for the community to consume. The idea is to use cheap IAX and SIP soft clients to hold most of the calls. Bandwidth is not free (as in beer) but it is much cheaper than the dial-in conferences.

So toward these ends, I've spent some time setting up an Asterisk server on our network and getting a reference implementation working for internal testing. That has all worked out quite well so over the next month or so we on the IT team will be working on the rollout plan. Project leads and PMC members will get a new component on the Portal which lets them schedule conference calls. It won't start out doing anything fancy like managing invitations, but it will let people schedule conference times and get a conference number and pin. Calls are only valid for the time during which they are scheduled. Scheduling will be live, which means you can schedule a call for immediate use. Here's what the panel will look like for showing existing conferences:

If you have a few scheduled they will all show up here. Old conferences will only show up for a few hours and then disappear. If you want to add a conference, the component tries to be somewhat intelligent about what you want to do and will automatically pick a conference number, PIN, start time, and duration. The default start time is immediately, but it can easily be scheduled for another time (times are in America/Montreal timezone, otherwise known as Eastern Time). There is no fancy date picker at the moment, but we have some ideas on how to make this easier to use. But for now, here is what the "add" panel looks like:

Nothing too amazing in the implementation, but this should really make life easier for projects who need to have calls for people across diverse locations.

Look for it in the near future!

Tuesday, June 12, 2007

Europa test-drive - part II

Yesterday I took Europa (Eclipse 3.3RC3) for a spin, and it proved to be a great experience for me. Finding and installing additional tools for Eclipse is much, much easier than before, thanks in part to the Europa discovery site. Today, I'm trying the experiment again, except this time I'm going for the latest Webtools bits. To help me do my Web work, I also need the ECF components, as well as the Datatools SQL tools. I'll be going with Eclipse 3.3 Release Candidate 4, which was uploaded by the platform team yesterday morning.

The story starts the same as yesterday, so I'll skip to the fun. In the Europa discovery site, I choose to install ECF, and upon picking the MySQL enablement package, I am told that there's a missing requirement - "org.eclipse.datatools.modelbase.feature (", or compatible. The packages are listed using a descriptive name, not the plugin name, so I cowardly click Select Required. Problem solved, easy stuff. As was the case yesterday, I see the scary "this plugin is not signed" dialog for most of the plugins. Come on guys, signing code is easy and free!

After the workbench is restarted, that's where I get the shocker with the new welcome screen:

1. "Eclipse is a kind of universal tool platform - an open extensible IDE for anything and nothing in particular". What does that mean? I hate that tag line, and we got rid of it on our website a long time ago. I've opened bug 192180 to request it be updated to reflect the times.

2. I'm guessing this is ECF's doing - thanks for putting that there, it should help me get going with ECF.

3. Java Development? It was installed as a required plugin. Makes sense, since I installed Webtools.

4. Wasted space - I installed Datatools and Webtools, perhaps they could have added a Getting Started with ... to help me get going?

Upon closing the welcome screen, I realize that the Datatools and Webtools plugins don't seem to be there. My first guess is that I'm not running Java > 1.4.2. Although Eclipse itself only requires 1.4.2, some add-ons require Java 5. John suggests we recommend a JRE 5 on our new downloads page, but a Failed Dependency warning would be neat too.

Monday, June 11, 2007

Europa test-drive

Europa is only 18 days away, so this morning I decided to take it for a spin. Normally I'd download and install Eclipse for PHP development, but the PDT project isn't participating in the Europa release this year, so I'll simply pretend that I'm a hotshot C++ developer who wants to get going with the CDT.

I start by downloading the Eclipse platform runtime, using 3.3 RC3 - just about the smallest bare bones Eclipse you can get. Upon launching Eclipse, I am greeted with the Welcome page. I go to What's new and click "New updates" and choose the Europa discovery site.

I'm prompted to select a mirror site in the USA -- darn, no Canadian update mirrors -- then I pick my projects. The breakdown of projects and categories is quite descriptive: you can't go wrong with "C and C++ Development". I also pick the Remote Systems Explorer and the CVS client.

After downloading the bits, I'm warned that the CDT plugins aren't digitally signed. It would be nice if they were, and I have a feeling they will be for RC4, as Doug inquired about JAR signing last week. The RSE plugins were signed, although the dialog requires me to validate the raw elements of the certificates (couldn't Eclipse show this in a cleaner format?)

Everything seems valid enough, and with everything installed, the workbench is restarted and I switch to the C/C++ perspective to create a new project. I base it on the Hello World example that's already provided, as this pretty much matches the extent of my C knowledge, and run a build. Within mere seconds, I have a 32-bit ELF executable sitting on my hard drive.

Lately, Eclipse has been making great strides in providing an easier download and install experience for the user, and although this is not a particular goal of Europa, it succeeds in this area by providing a discovery site loaded with projects and plugins to help users get going without needing to fetch and install plugins manually. The Packaging project will make this process even easier, by creating standard Eclipse packages (or distributions, in the Linux world) containing plugins for the most popular combinations of Eclipse tools. Great stuff indeed.

Thursday, June 07, 2007

Bugzilla: search for attachment creator

Jerome opened bug 191194, requesting the possibility to search for attachments that were submitted by someone specific. Although Bugzilla's Advanced Search is very thorough, it doesn't include such criteria. But after a bit of digging, I found this little gem on Mozilla's bugzilla, which suggests this workaround:

In advanced Search, pick Attachment Data / Changed By / [email address of Attacher]

I tried this, using my own email address, and it listed all the bugs where I created an attachment. Cool.