Denis Roy | Eclipse Webmaster

Authentication changes at Eclipse.org

2012-01-05T15:36:00.002-05:00

As you may have heard, we've changed our authentication scheme at Eclipse.org. Instead of having Committer accounts in one database and Bugzilla users in another, both of those databases have been brought together.

We did this for many reasons: it felt strange to ask users to create a "Bugzilla" account to be able to participate on Forums; it was confusing; our sites, such as Wiki and Marketplace, needed to keep track of two auth sources; Eclipse Committers essentially had two accounts, which made no sense; for every new software tool we'd install, we needed to write custom plugins to allow authentication against the Bugzilla database.

Now everything is consolidated, and although we don't have a complete Single Sign On solution, this change has paved the way towards that. It was also a required change in order to deploy the Gerrit code review system.

As with any migration, it wasn't without its bumps and glitches, but overall it went well. Thanks to everyone who tested, re-tested and wrote in to report problems and bugs.

Meeting the people is what this is all about

2011-11-25T09:40:00.004-05:00

I attended the Ottawa DemoCamp yesterday, where a bunch of Ottawa Eclipse community members gathered for a bit of food, a bit of beer and a lot of cake to celebrate Eclipse's 10th birthday.

These informal gatherings are not only about food, beer (and cake) and idle chat -- there is some quality learning happening at these events, and a great opportunity to meet some interesting people.

Yesterday I made some new friends who work on Eclipse technologies for the federal government of Canada right here in Ottawa. The Department of National Defense, no less (yes, Canada has one of those).

I also had a short but interesting talk with Ken Hussey about his work on Puppet-related tools at Cloudsmith. I've had Puppet on my list of tools to investigate for quite some time now, and seeing Cloudsmith in that space may help precipitate things.

A gentleman named Antoine also talked to me about Sonar -- a code quality and analysis platform. He made it sound really good, and the screenshots make it look even better. Definitely on the radar.

Ericsson was also in the house, and I had a short conversation with someone whose name I unfortunately cannot remember. But she was very nice, and explained that they use Eclipse and the CDT. I wanted to introduce her to Mr. CDT ("I pity the fool") himself, since he was also present yesterday -- but alas, that never happened.

Lastly, being seen in public with such local celebrities as Kim Moir, John Arthorne and Pascal Rapicault is always a great way to boost one's image.

Cheers!

Crowdsourced translations make Eclipse in English better too

2011-11-15T09:35:00.002-05:00

A few years ago I actively worked on the Babel project -- a crowdsourced solution to Eclipse globalization. Since then the translation tool has been doing its job -- allowing the community to contribute translated strings, which are then turned into downloadable language packs.

Lately, one of our contributors in Japan has been opening a series of bugs outlining issues with the English strings in the message files. Pure awesome.

You have to love the power of OSS -- the more eyes we have looking at code, the better it becomes.

My first EclipseCon Europe!

2011-11-02T08:24:00.003-04:00

I'm in Germany this week for my first EclipseCon Europe. I had heard rumors that the Wifi's performance wasn't comparable to that of EclipseCon NA so I came here to help.

Just to be clear: I'm not involved in the network or Wifi setup at either conference, but I do bring a copy of download.eclipse.org with me. I then set up a mirror and some redirects so that your Eclipse bits aren't sent over the Internet.

This morning was a typical EclipseCon Day 1: although the conference Wifi is great, as attendees start up Eclipse, the p2 downloads pile up against the poor Internet connection, making it nearly impossible to do anything else. But after some caches, redirects and tweaking compression on www.eclipsecon.org, the local mirror is putting out over 800KB/sec and the rest of the world feels accessible again.

I'll keep monitoring the situation throughout the conference and tweak content as I can to help make sure your Wifi experience at econ2011 is the best it can be.

10 years of Eclipse as seen by /etc/passwd

2011-10-28T14:02:00.004-04:00

If you've worked on a *nix system, you're likely familiar with the /etc/passwd file -- the main list of accounts on that system. Although eclipse.org does not use the passwd file for user accounts, examining the compatible getent passwd tells the story about who was involved in setting up the original dev.eclipse.org way back in 2001.

Keep in mind that we delete accounts of committers who have been completely decommitterized; therefore, the people you see mentioned below are still active participants in the Eclipse community even after 10 years.

First up, the standard root, mysql and other system accounts, then here's what I see:

500:kmoir

I wasn't there in 2001, but judging by her UID=500, I'm guessing Kim Moir was the one who inserted the first Linux CD in the drive and installed a computer called "dev.eclipse.org".

Next up:
512:mmartin

Mark was a webmaster@eclipse.org for a long time. Although he and his account are no longer active on dev.eclipse.org, he gave me my webmaster training at the IBM Queensview lab back in October 2004. So his account stays for my own nostalgia :-)

Judging by Mark's UID=512, I'm guessing a few other webmasters were blessed with accounts on dev.eclipse.org before his.

Next up:
1000:droy
1002:jeff

I am an imposter in that list, since Jeff McAffer predates me by a few years. But I have root@eclipse.org, so I faked my UID=1000. Deal with it :-)

After creating Jeff's account, Kim wasted no time getting things set up:
7003:cvs
7004:viewcvs
7005:anonymous

The jump from UID=1000 to UID=7003 is likely an artifact of an IBM security policy for user ids, since there is nothing below 7003.

Then, we have the first Eclipse committers:
7008:pmulet
7012:mcq
7020:dj
7026:johna
7033:ggayed
7034:veronika
7036:steve
7037:silenio
7043:dmegert
7047:daudel
7048:jeromel
7050:oliviert

Although some of them are not very active anymore, some of them still are, but one thing they all have in common is that they helped shape Eclipse as you see it today.

Mailing lists on dev.eclipse.org came a bit later in 2002 with the creation of a 'mailman' account, followed shortly by a 'webmaster' account. At that time, I'm guessing the real webmaster became a few people who perhaps occasionally shared the job.

7336:david_williams

David Williams and some of the still-active WTP committers are listed next, which likely happened in 2003-2004 when the WTP top-level project was created. Today, David Williams is a name recognized more than even Eclipse itself.

A bunch of BIRT committers are listed in succession, which matches up to 2004's creation of the top-level project of the same name.

8249:mmilinkovich

Although Mike was on the Eclipse scene long before I was, he only got an account on dev.eclipse.org when the Phoenix project was created and he became a committer. Bye Bye Frames!

The Eclipse superstar with the most letter Z's in his name... circa 2005 ... followed by our own Conference Queen Anne Jacko.
8444:caniszczyk
8636:ajacko

I could go on... But I wanted to highlight the first years. Besides, I must be the only sysadmin dumb enough to put user id's and UIDs in plain text on a blog.

Fast forward six years... and we're up to UID=9826. Who will have lucky account UID=10000 ?

Hudson building Hudson

2011-10-19T15:56:00.002-04:00

Ok, so I'm easily impressed. I think it's cool.

30.8 Terabytes!

2011-10-03T14:15:00.006-04:00

Last month, Eclipse.org servers have moved 30.8 TB of data to the Internet. 30.8 Terabytes! That is a first -- even during the yearly release, we've never moved so many bits in one month. And September was a short month :)

It's been a steady climb since we started tracking bits in 2006, but it appears that since early 2010, things have really taken off like a rocket.

So I decided to compare our monthly bit throughput with the size, on disk, of the Eclipse code repos.

Code size is increasing quite linearly, with a noticeable boost in size midway though 2009.

Next up is the same bit throughput and SCM size on disk, compared to our binary downloads footprint.

As we prepare for Eclipse's 10th anniversary next month, these numbers show that there's no slowing down the momentum that Eclipse has gained during that time. Cool stuff!

Going to EclipseCon Europe: is there enough beer in Germany?

2011-09-23T10:20:00.000-04:00

In the years past, I was told I could not attend EclipseCon Europe because there wasn't enough beer in Germany to satisfy this thirsty Canadian. I'm not sure how much truth there is to that, but it's what I was told.

Anyway, as the European edition of my favourite gathering has grown, so has the need for some official IT support. I'll be flying in Tuesday, Nov. 1 to help (however I can) with WiFi setup and to set up a download mirror like I do at EclipseCon North America. The download mirror is a real Internet saver -- these conference venues typically don't have tons of Internet bandwidth, so bringing a local copy of all your favourite Eclipse bits helps a lot. I look forward to seeing all my European friends, and to making some new ones!

In the interest of saving thousands of dollars in airline fees, I'll be staying over Saturday for some local sightseeing. Since this will be my first time in Germany, if you have any hints as to what I should do during my stay, please post up in the comments.

Edit 9/23: Apparently comments were disabled on my blog. Sorry 'bout that.

Indigo early access to Friends of Eclipse

2011-06-21T10:14:00.002-04:00

An email was sent out to the Friends of Eclipse, announcing early availability to the Indigo bits. Less than 2 minutes later, here is the resulting spike in usage on the Friends of Eclipse Mirror:

If you're a Friend, you can access the download links right now. If not, consider becoming a Friend today as part of the Indigo 500.

Torrent files are also up for early access to our p2p users.

Anatomy of an outage - part II

2011-06-09T09:59:00.002-04:00

Yesterday I posted about the first of two outages we've experienced last week. Today I'll post about the outage we had late in the early afternoon of June 2.

Similarly to our previous outage, we suddenly lost our ability to talk to our primary switch -- a Cisco 2970 24-port Gigabit switch, in service since October 2004.

We feared the worst -- a tripped power circuit caused by a faulty power supply. Again? Not this time -- the switch was simply 'frozen' with an orange alert lamp. After cycling the power, we were back in business.

But why did it just freeze? Was it beginning to show signs of fatigue? Matt and I took no chances -- with enough available ports on our much newer 48-port Cisco 2960 (part of a hardware donation made by Cisco for EclipseCON 2009), we migrated all the connections off to the new switch.

Accurate graphs and documentation allowed us to migrate port settings, VLANs and QoS rules quickly to the 'new' switch.

We need cool screenshots of Eclipse

2011-06-09T09:05:00.004-04:00

Have you looked at http://www.eclipse.org/screenshots/ lately?

Yes, it is old.

I was thinking that we could move that page to the Wiki (perhaps http://wiki.eclipse.org/Screenshots ?) so that everyone can submit cool-looking screenshots of Eclipse in action.

If you agree it's a good idea, I need your help to get started. If you create the wiki page and upload/post your Eclipse In Action screenshots, I'll set up redirects from the old page to the new page.

Anatomy of an outage - part 1

2011-06-08T09:54:00.004-04:00

It happens to Facebook. It happens to Hotmail. And yes, it happens to Eclipse. Last week Eclipse.org suffered two distinct outages. Today I'll discuss the first one "briefly".

From a bandwidth point-of-view, this is what a normal 24-hour period looks like.

This is what we saw on the morning May 31:I had just walked into the office when I discovered we were completely offline. Since the Foundation's Ottawa office is wired directly into the Eclipse.org switching gear, I was surprised to discover I couldn't even talk to our switches. So I hopped in my car and drove to the Data Center, which is about 10 minutes away.

Turns out a few servers were powered off, as was our main switch and firewall -- the circuit breaker had tripped. When the technician reset the circuit, sparks spewed out of a power supply in our DS4 raid array. Not good.

We pulled the faulty component, reset the breaker and restored power to the switch and firewall. "Problem Solved" I thought. Not so fast.

For reasons unknown, our primary NFS server (which hosts shared files for CVS, Git, SVN, and many of our websites) was frozen. I found that strange, since they do not share the same power circuit. Heck, they aren't even on the same voltage rating. This specific server doesn't normally output video, relying instead on a Hardware Management Console (HMC) to communicate with the operator.

As luck would have it, the HMC was one of the servers connected to the downed power circuit, and filesystem errors were preventing it from coming back online. So I had no insight as to what my faulty NFS server was doing, or why it was frozen.

After a bit of waddling and trying to restart it, I decided to abandon the primary NFS server, and notified fellow webmaster Matt to begin the failover process to the secondary server. That is when services began recovering, about 1h45 minutes later.

To this day, we still haven't figured out what is wrong with our primary NFS server, although Matt has fixed the HMC. And the DS4 storage box is still operating on only one power supply (although another is on order and is expected to arrive soon).

We've decided to begin the process of acquiring new NFS server hardware. The 'old' servers are big, complex machines that have been in service 24/7/365 for almost 7 years. They have served us well, but they are showing signs of age, as memory failures, disk failures and backup battery failures are manifesting themselves more frequently.

Stay tuned for some explanations on the other complete outage, that came upon us just days later...

Eclipse Forums upgrade gone bad.

2011-04-26T11:07:00.002-04:00

A few weeks ago I set out to upgrade our aging Forums software. We're about 1.5 years behind the latest release, and with numerous bugfixes and new features, an upgrade seemed compelling. For instance, here are some of the things we're missing:

Per-forum search. Because of the size of our forums (650,000+ messages), the single database server would be crushed by the search queries.
Single database server. The current version only supports a single server, limiting scalability and preventing us from enabling search.
Broken NNTP bridge. Our forums are linked to the NNTP newsgroups, and attachment import/export is broken, as are many extended character sets.

Despite my best efforts, the upgrade failed miserably and I reverted back to the old version. The software we use, FUDForum, is a decent piece of code, but it has a rather unorthodox way of doing templates and themes, which leads to it being easily broken on our rather restricted multi-server PHP environment. When I set it up almost 2 years ago, I had to twist and contort it to fit. The result was a brittle piece of software that did what it is supposed to, as long as you walked on eggshells when managing it.

On the bright side, a brand-new install of the same software works like a charm on eclipse.org, without any of the previous twisting and contorting needed, and it simply feels more robust. I guess 2 years of development will do that to software. There's just something with the upgrade process + our hacked version that simply doesn't jive.

All told, we cannot simply continue using the current version. It works, but with many broken parts, and with many security holes now patched, it needs fixing. I can think of two options:

Option 1: Move to a different forum software. This would break links to all the existing messages, present a new UI to our users and would put us webmasters in unfamiliar territory by discarding the two years of knowledge we've accumulated with FUDForum.

Option 2: Install the new version from scratch, and re-import all the NNTP messages. This would also break links to existing messages, but it is known to work on our servers, and FUDForum (although not perfect by any means) is now well understood by the webmasters.

What is your take? Feel free to discuss with us on bug 342708.

EclipseCon poll results

2011-03-18T10:16:00.003-04:00

On Wednesday I asked you a few questions about your EclipseCon experience.. Here are the results:

39 this.is.my.first.eclipsecon.ever.its.going.to.be.great---they.say.you.buy.beers

Welcome to all the newcomers! Look for me at the bar and I'll buy you a beer *

36 please.set.up.a.better.404.page.for.eclipsecon

Popular option... Ask and you shall receive. Go look...

30 ive.been.to.many.eclipsecons---why-have-you-never-bought-me-a-beer

Sorry if I've missed you... Hopefully I'll do a better job this year *

27 ive.been.to.all.the.eclipsecons---and.I.know.you.buy.beers

Yes, I know who you are... Quite a costly bunch :)

25 ive.been.to.a.couple.of.eclipsecons---I-prefer-to-run-than-drink-beer

If running wasn't so early in the morning, I'd likely join you. I'll try this year -- I might actually make it a few feet.

Some other responses and comments I've received:

webmaster.not.going.this.year.will.miss.the.beers

That's really unfortunate -- there's always EclipseCon Europe, and EclipseCon 2012

webmaster.ive.never.been.at.eclipsecon---please-send-beer-to-france

I can't really send beer to France but here's what I can do: to the person who posted this, send me an email to webmaster@eclipse.org. If your IP address matches up to that comment in my Apache log, I'll send you some Eclipse SWAG!

webmaster.ive.been.to.all.the.eclipsecons---and.I.know.you.buy.beers----altough-thats-not-entirely-true----well-its-not-true-that-ive-been-to-all-eclipsecons-but-you-guys-always-bought-beers

I'm so confused :)

webmaster.i'd.buy.you.a.beer.if.i.were.going.but.unfortunately.i'm.not.(sad.smily)
webmaster.i.am.not.going.to.make.it--can-you-mail-me-the-beer?

It's too bad you can't come ... I hope to see you next year.

* Disclaimer: offer valid while supplies last, no rainchecks. On approved credit. Offer subject to change without notice. See newspaper for details. Professional driver on closed course, do not attempt. Viewer discretion is advised. Product may differ from illustration.

Is everyone ready for EclipseCON?

2011-03-16T11:14:00.005-04:00

Only five days until the big event begins (well, four days, if you consider the conference to officially kick-off Sunday evening at the bar).

If you want to distribute large files (or many files) to fellow attendees, upload your files to download.eclipse.org ASAP, since download.eclipse.org will be mirrored at the conference centre in its entirety -- including Nightly builds.

As per the tradition, here's a whacky webmaster poll to scope out the conference newcomers and veterans. Please choose one of the following options -- results will be posted tomorrow:

http://eclipsecon.org/webmaster.this.is.my.first.eclipsecon.ever.its.going.to.be.great---they.say.you.buy.beers

http://eclipsecon.org/webmaster.ive.been.to.all.the.eclipsecons---and.I.know.you.buy.beers

http://eclipsecon.org/webmaster.ive.been.to.many.eclipsecons---why-have-you-never-bought-me-a-beer

http://eclipsecon.org/webmaster.ive.been.to.a.couple.of.eclipsecons---I-prefer-to-run-than-drink-beer

http://eclipsecon.org/webmaster.please.set.up.a.better.404.page.for.eclipsecon

Eclipse makes refactoring fun.

2011-02-04T09:42:00.002-05:00

"Well, duh" you say.

For seasoned developers and code gurus, you already know that. But for server guys like myself, the thought of a major refactoring of code -- especially code not authored by self -- leads to nightmares.

I spent the last few days getting intimate with Eclipse Helios and the PDT working on bug 303756. To my own surprise, I am progressing remarkably well thanks to Eclipse's navigation tools, content assist, and the little things like CTRL+SPACE.

How to ask questions the smart way ... without so many words?

2010-11-16T10:11:00.000-05:00

The "How to Ask Questions the Smart Way" guide that was linked from our forums page is a great guide, but it has way too many words. Let's face it -- if someone is too lazy to use Google or pick the right forum, are they really going to read all that?

So I've started my own, very abridged version. Please help me make it better, but focus is on brevity. This document will be linked from the Eclipse Forums.

Eclipse.org user foums for non-Eclipse.org projects?

2010-11-11T09:05:00.000-05:00

This forum post got me thinking:
"Webmaster please create a "Python IDE" topic branch "
http://www.eclipse.org/forums/index.php?t=msg&th=200110

The Eclipse forums are intended for user discussion regarding Eclipse and
Eclipse.org projects. Of course, we don't want to be in the business of
creating hundreds of forums for all the non-Eclipse.org projects out there.
But it occurred to me that there are likely some projects out there that don't
have user forums; users can't easily seek support and are left to fend for
themselves.

Would it be a good idea to create a generic forum to allow users to discuss
non-Eclipse.org projects?

I've opened a bug to track the issue -- please feel free to comment there.

ECF moves to Git

2010-10-13T04:34:00.000-04:00

It seems traction for Git is slowly picking up here at Eclipse.org. This morning I got the "go" to switch the CVS repository to read-only mode for the ECF project, as the team moves their development completely onto Git.

Although a few smaller/younger projects have moved already, ECF is one of the first mature projects with a rich history of several years to make the move.

You can browse the ECF Git repo here: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/

Congrats to the ECF team for taking the plunge!

Better (some?) Wifi coverage at ESE

2010-08-24T10:31:00.000-04:00

I've never been to Eclipse Summit Europe. I'm told it's because there's not enough beer in Germany to accommodate my lavish consumption.

Regardless, I've started working with the conference organizers towards providing better/more Wifi coverage at the conference in a manner that would resemble what is provided at EclipseCon.

This is where we play the game of estimating how much Internet bandwidth you'll use up, double the number, then double it again, and then have our calculations be demonstrably wrong once the conference is in full swing. But that's part of the fun!

This is why I love Open Source Software

2010-08-12T09:35:00.000-04:00

As I prepare to replace dev.eclipse.org with new servers, I noticed that IPZilla, a tweaked version of Bugzilla, wasn't working on the new servers. It was bound to happen -- an antique version of Bugzilla running on the latest Perl.

Googling the error, I stumble upon this 3-year-old post from Max-Alexander Kanat, who responds to a user who is trying to run Bugzilla on a development version of Perl. For that person, downgrading to the stable Perl works, but I can't (and don't want to) do that.

But Max didn't stop there. That post raised a red flag, and he diligently opened a bug to investigate. Patch issued, RESOLVED/FIXED.

I applied the one-line patch to our antique system, and voilà. RESOLVED/FIXED for me as well. 15 minutes from start to finish.

I love Open Source Software.

Forums and NNTP newsgroups are back online

2010-08-03T12:58:00.000-04:00

If you've been missing the forums/newsgroups, you'll be glad to know they are back online now.

Winner of the WTF/loc award

2010-07-29T04:20:00.000-04:00

To gauge the quality of code, we've all seen the metric called WTF/minute.

I'm inventing a new metric, called the WTF/loc, based on this very short snippet of code.

#!/usr/bin/perl
use strict;
my $cmd = "perl " . shift;

my $h = <STDIN>;
while ($_ = <STDIN>) {
        open (S, "| $cmd");
        print S $h;
        print S;
        for (2..1000) {
                last unless $_ = <>;
                print S;
        }
}

A few comments to explain what's going on would have made this snippet of code much more maintainable.

Making servers go faster

2010-07-20T06:56:00.000-04:00

In computers 101 they teach you that computers can often go faster with more RAM to avoid paging, and to cache files from snail-paced hard drives.

Eclipse.org gets most of its servers from generous donations by companies like Intel and IBM, but often* the servers lack a bit in the RAM department. When Google made a donation of actual money, we knew exactly where to spend it.

By putting tons of RAM in our front-end Web, CVS, SVN and Git servers, your requests have a good chance of being fulfilled without going to backend servers or, much worse, disk storage.

Here are some of the comments I've seen so far, and we're not even done with the upgrades yet -- we still have to upgrade the Git servers, and dev.eclipse.org for CVS and SVN.

epp-dev: "I've never seen an EPP build finishing so fast. This time it took only 3.5 hours. Compared to 7 to 8 hours that's quite fast. It looks like there is no I/O on build.eclipse.org that causes CPU wait cycles."

Eclipse/Helios Retrospective: "Bugzilla performance is much better"

Webmaster Inbox: "CVS is very fast today"

Many thanks to Google, IBM and Intel for the recent donations, and to AMD, Cisco, HP and IBM for donations made in the past.

* Not always. The latest servers donated by IBM came packed with 36G of RAM!

What's new at Babel?

2010-07-09T10:49:00.000-04:00

The Babel project (crowd-sourced localization/translation) has been a bit quiet lately. Although there's not much happening on the development front, the translations keep coming in. Eclipse 3.6 (the base platform for Helios) is over 85% translated for 7 languages (Simplified Chinese, French, Japanese, German, Korean, Traditional Chinese and Spanish). We have over 72% translation for fifteen other languages, including Arabic, Greek, Polish, Dutch, Italian and Portuguese.

We will be releasing a Helios language pack soon. If you just can't wait, you can get one of our Nightly builds.

The Hero of Helios (and the crazy woman behind him)

2010-06-23T06:11:00.000-04:00

There is little doubt that David Williams is the Hero of Helios, but I often wondered why he invests so much of his time into the annual release.

Then it all became clear that there was a persuading force behind him all along.

Thanks David for everything you do.

Helios is here!

2010-06-23T05:48:00.000-04:00

By now you've heard the news about Helios. People are talking about it all over the place: blogs, news, Twitter (not that I use Twitter, but Ian tells me). People sure like their Eclipse wares -- even the Wanted: 360 Friends of Helios has busted out of the screen. Congratulations to all the teams involved!

On the webmaster & servers front, there are a number of things that has made this release even easier than in the years past:

- No last-minute rebuilds. Kudos to all the teams for bringing together so many projects into a cohesive package. It is no small feat, and despite having witnessed the effort four times before, the amount of co-ordination required to pull this off is truly impressive.

- New hardware. IBM, Intel and Google stepped up and donated hardware to handle the load of all those May-June builds. During the actual release, our web servers were handling over 5000 connections without breaking a sweat.

- Early syncing. EPP packages were sent to our mirrors over the weekend, which is the best possible time (bandwidth-wise) to do this. Kim sent the Classic/SDK bits several hours earlier than planned too, which helps avoid the sync rush we usually experience when all our high-bandwidth mirrors come knocking on our door.

- Gigabit Internet connection. With some clever tweaking, I was able to double, even triple available bandwidth for extended periods of time, without incurring any cost to the Foundation.

Checksum poll draws clear conclusion: I need a new poll mechanism

2010-06-18T11:10:00.000-04:00

Earlier this week I launched another of my ridiculous polls to find out what your stance is on download checksums. Here's what I found out: the overwhelming winner was who-needs-them-just-download-and-run

MD5, SHA1 and PGP are your favourite methods, in that order. Strangely, I got a few hits for a totally invalid response: only-when-I-DL-from-eclipse-dot-org-dont-trust-those-webmasters. Shocking!

Of course, even with a radio-button-type of poll, folks feel compelled to hack my system and fill my Apache logs with senseless commentary ;-)

can-i-trade-them-for-beer: Checksums are free as in beer, so technically, we've already traded. Or there is nothing to trade. I'm confused now.

i-know-i-should-use-them-but-my-downloads-run-fine-most-of-the-time-and-eclipse-releases-are-signed-anyway: The Eclipse "release" isn't signed -- the individual JAR files inside are signed. As far as I know, the "eclipse" executable (eclipse.exe for you Windows fans) isn't signed...

Then the typical comedians:

votes-are-funny-but-I-get-always-these-404-File-no-found-error-pages

you-forgot-to-add-checksum-to-the-survey.fb3a6ba5ac1ca4c88fc82bf919155baca5b05f06

checksums-are-important-I-always-check-it-by-mental-arithmetic

So there you have it. For those who feel checksums are important, I've incorporated a mechanism to display checksum links for very file that asks you to pick a mirror:

Helios Desktop Wallpapers

2010-06-17T06:29:00.000-04:00

We need your help designing a desktop wallpaper for Helios. If you are creative, have a look at the various Helios web pages and attach one (or many) Helios desktop wallpaper(s) to bug 312811. As you can see by the bug, Lynn has already submitted her wallpaper. While her graphics and artistic skills are hard to beat, I'm sure some of you can come up with something nearly as good. Maybe.

Quick poll: when you download something, do you check your checksums?

2010-06-14T11:28:00.000-04:00

Here's a quick poll: when you download software from the Internet, do you run a check to make sure that what you have downloaded is what the publisher intended for you to download? Hit one of the links below to cast your vote. You will get a "Page Not Found" and that's OK -- I'll compile the results from the Apache logs. Internet polls don't get any less accurate than this!

http://eclipse.org/checksums-are-important-I-always-check-with-md5

http://eclipse.org/checksums-are-important-I-always-check-with-sha1

http://eclipse.org/checksums-are-important-I-always-check-with-pgp

http://eclipse.org/checksums-who-needs-them-just-download-and-run

http://eclipse.org/checksums-what-are-they-and-why-are-you-here

I'll tally up the results towards the end of the week.

Helios, June 23: Beat the rush, become a Friend

2010-06-14T05:11:00.000-04:00

Helios, based on the upcoming Eclipse 3.6, will be released next week -- June 23 to be exact. But Friends of Eclipse can get it a day early and beat the release day rush.

On June 22 an email will be sent out to all our Friends with direct links to all the new Helios bits. Being first in line couldn't be easier!

Great websites @eclipse.org

2010-06-09T06:50:00.000-04:00

Once upon a time, an Eclipse project website was a second-class citizen -- an afterthought, something that was not very important. But not anymore -- projects are turning their websites into great resources for the community, and they're looking good doing so. Here's a few recent examples:

The ATL project use the new default template (provided by the Xtext team) and add in their own touches. Very classy.

The Acceleo site is pretty much the same, except the screenshot is replaced by a cool screencast.

Webtools uses the same basic layout, but incorporates an audience focus and some very professional-looking graphics.

It's nice to see all the great work done on these websites!

RESOLVED/LATER != RESOLVED/NEVER

2010-05-31T05:01:00.000-04:00

Who says old bugs never get any attention? https://bugs.eclipse.org/bugs/show_bug.cgi?id=4922

The number of antique bugs still open is not very big:

mysql> select count(1) from bugs where bug_id < 10000 and bug_status in ("NEW", "ASSIGNED", "REOPENED");
+----------+
| count(1) |
+----------+
|       77 |
+----------+
1 row in set (0.01 sec)

Eclipse.org gets new database servers

2010-05-26T04:53:00.000-04:00

Last weekend we've migrated our MySQL master and slave databases to the newly donated hardware from Intel and Google. In this case, both servers are Intel SR-2625's with two Xeon e5540 quad-core CPUs and 48G of RAM (courtesy of Google). The servers also have a nifty RAID array, but the strategy was to avoid touching the disks. With 48G of RAM, mission accomplished.

Check out the load average for yesterday:

That's a peak of 0.87 for dbmaster, and 1.98 for dbslave, with averages of about 0.50 each. With 8 CPU cores, this kind of load average is just barely above idle.

Previously, the databases coexisted with the NFS file servers, so there is an inherent double-whammy in performance gains by removing MySQL burden from the NFS servers.

I don't see emails like this everyday.

2010-05-18T05:31:00.000-04:00

An email with the subject line "CVS is very fast today" gets our immediate attention.

Over the weekend we changed a few mounts on our NFS servers to spread the load a bit better, and to more closely match our classes of service. As a result, our primary NFS server, which serves files for committer CVS, SVN, Git, and other has-to-be-fast content has seen a drastic reduction in disk load.

Committers should expect drastic improvements in performance to their favourite code repos.

Ewwwww... Get it off of me.

2010-05-14T03:50:00.000-04:00

I bought a copy of Windows 7 Professional yesterday to install it on one if the newly donated IBM servers we'll use for testing. I cried during the entire transaction.

It's not that I don't like to pay for software. I just don't like paying for Windows. Yuck.

When I got home I took a cold shower, and showed my 4-year-old how to write shell scripts.

Happiness is keeping it in RAM

2010-04-20T08:57:00.000-04:00

Last week I deployed a new download.eclipse.org mega-machine. If you haven't noticed, that's great. The transition was completely transparent.

Reduced disk seek time on our file servers by maximizing file cache hits is what this is all about. Here's a bandwidth graph of what our download server is sending to the Internet -- over 70 Megabits/sec (the green line) during the Eastern timezone daytime.

In return, this is what the download server is fetching from the NFS file server (the blue line). You can see that as the server came online last week, the file cache was empty, and all the download requests needed to be fetched from NFS. But today, NFS is serving less than 5 Megabits/sec to the download server.

70 Megabits out, 5 Megabits in. Good cache hits there.

But in practice, since our bandwidth is limited, you won't see a faster download speed. But your download should start instantaneously. However, the biggest gain is achieved for services that need to use the NFS file server, such as CVS, SVN and Git.

But this is only the beginning ... wait until you see what we're doing with the MySQL database servers...

Setting up a download server.. How much RAM do I need?

2010-04-12T09:50:00.001-04:00

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

If you've been following along, you know that I received a bunch of hardware to upgrade Eclipse.org. 'Keep it in RAM' is my thinking with all this. Currently, our servers spend a lot of time waiting for disk resources, so the benefit of keeping requests in RAM is twofold: cached files are served quickly, and disk resources are freed for faster access to disk.

Right now I'm in the process of setting up a high-capacity download server to replace download.eclipse.org, so the question is -- how much RAM do I need?

I started by examining the combined Apache logs of download.eclipse.org for any given day.

zegrep "GET .* 200 " download.eclipse.org/access_log.3.gz | awk '{print $7}' > filelist
filelist contains 2,439,051 successful "GET" requests

sort filelist | uniq -c | sort -nr > filelist-sorted
filelist-sorted contains 73,184 entries

So each day our server only reads about 75,000 files -- but serves them 2.4 million times. There is a huge potential for cache hits right there. With a small perl script, I gathered the size (on disk) of those 73,184 files. Total: 25G.

#!/usr/bin/perl

open(FILE, "<filelist-sorted");
while(<FILE>) {
        $_ =~ s/\n//;
        ($c, $f) = split(/ /, $_);
        $size = -s "downloads" . $f;
        print "$size $c $f\n";
}

So if I get 24G of RAM, I'm sure that most of the file requests will come from cache. Actually, if I load up the numbers in a spreadsheet, I see a wicked long-tail distribution. In fact, 134 files are fetched at least once each minute and account for 43% of all requests. If you consider the RAM requirements of the OS and Apache, 24G would be great -- for today's needs. What about next year?

Considering it's cheaper to buy RAM when it's popular (commodity), I put 64G of RAM in the new download.eclipse.org. It should be more than sufficient to hold the entire week's worth of download files, file attributes and such while keeping disk requests to a minimum. It will also have plenty of RAM for the OS and Apache, even when things heat up in June.

We're also moving to the Apache worker MPM for download.eclipse.org. It is multi-threaded, so with high client counts, it consumes much less memory than the prefork model. PHP files (which are only a very small fraction of the hits) will be served over FastCGI.

So there you have it. Your downloads won't necessarily be faster since we are limited by bandwidth, but they should begin faster. The new setup will also free disk resources for those files that cannot always be cached, such as CVS, SVN and Git. Win-win!

Git vs. IP provenance -- DVCS with a twist

2010-04-01T05:35:00.000-04:00

With a Distributed Version Control System (DVCS) such as Git, there is no reliance on a central system. This allows developers to easily pound out and exchange code amongst themselves.

This also causes an Intellectual Property (IP) nightmare for OSS foundations like Eclipse, where IP and IP provenance are keystones to the organization.

For a DVCS to work at Eclipse.org, we need to make some compromises. The Eclipse Foundation needs to know where the code comes from, and who wrote it. Picture this:

In the case above, Jimmy and Jina are not known to Eclipse.org, but they are working together to help Billy, an Eclipse committer, by contributing code. Since Git does not maintain a "push" log, if Billy's push to git.eclipse.org was to succeed, we would have a git commit history that would contain entries of people that we do not know. In fact, since the email address is configured by the user, Eclipse has no real way of determining the validity of the information.

In the above case, the push will fail. So how does Billy make this work?

The correct workflow is seen above. Jimmy and Jina implement a feature and wish to have it included in Eclipse. They submit a patch to Bugzilla, where it is then reviewed by a committer by following the Eclipse Development Process and Due Diligence guidelines. From here, the committer will submit the code contribution to IPZilla if need be.

If the code passes all the required reviews, Billy incorporates it into his local Git clone and pushes the code to Eclipse's Git server. Since the Git commit log contains an entry for the Author name/email, Committer name/email and a commit comment, the names of the original authors, as well as the bug number can all be referenced for easy traceability.

This process does remove a bit of the "D" in DVCS, but if we follow a few simple rules, we can still take advantage of the benefits of Git while maintaining a clean IP trail.

EclipseCon: sometimes it's the small things

2010-03-22T08:19:00.001-04:00

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

Sometimes it's the smallest things that can make a conference feel so much better. The conference has only been "on" for two hours, and here's what I've noticed:

Breakfast: Fruit! Plenty of fresh fruit. A cereal bar. Pastries. But fruit!

Coffee: large cups that can hold more than 2 sips. Folks can actually grab a cup and walk away from the fountain.

Centralized: Everything is contained within the hotel itself, so you don't spend half your day walking around looking for the rooms. Much easier to meet up with friends and colleagues.

Props to the conference organizers who keep finding ways of improving an already great conference.

Got RAM?

2010-03-19T09:42:00.001-04:00

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

This is what 244 gigabytes of RAM looks like:

Thanks, Google!

Poll results: Why do you go to EclipseCON?

2010-03-19T04:39:00.000-04:00

In yesterday's poll, I asked you what were your primary reason(s) for going to EclipseCon. Here are the results:

41 /i-go-to-eclipsecon-to-meet-up-with-friends-colleages-and-to-network
39 /i-go-to-eclipsecon-for-the-free-beer
30 /i-go-to-eclipsecon-for-the-technical-stuff---talks-tutorials-bofs-its-all-good

It's a two-way tie for first place: you go for the great technical content and for the social/networking aspects of the conference. I'm sure the high number of "free beer" responses are just a statistical error, nothing we need to pay attention to really. Or, maybe it's a symbolic thing -- Eclipse is free as in beer, so you go for the free "beer". Yeah, that's it.

24 /i-go-to-eclipsecon-for-the-free-food
23 /i-go-to-eclipsecon-for-the-free-stuff
20 /if-its-a-link-i-just-have-to-click-it--what-is-eclipsecon-and-how-can-i-download-it

I'll let you decide what to make of those numbers.

10 /i-go-to-eclipsecon-for-downloads--much-faster-than-download.eclipse.org

Well that's interesting.

Of course, others felt it appropriate to formulate their own poll responses:

1 /i-go-to-eclipsecon-for-the-free-beer-from-the-webmaster

Nonsense. I honestly don't know what that person is talking about.

1 /i-wish-I-could-go-but-boss^H^H^H^Hwife-won't-let-me
1 /i-dont-go-to-eclipsecon-but-i-really-wish-i-did

I can feel for these poor individuals that can not attend. Maybe we'll see you next year?

1 /i-go-to-eclipsecon-to-meet-denis

I was really touched to read that one. I really was. Until I read the next responses:

/i-go-to-eclipsecon-to-meet-denis-and-wayne
/i-go-to-eclipsecon-to-meet-denis-and-wayne-and-ian
/i-go-to-eclipsecon-to-meet-denis-and-wayne-and-ian-and-donald
/i-go-to-eclipsecon-to-meet-denis-and-wayne-and-ian-and-donald-and-lynn
/i-go-to-eclipsecon-to-meet-denis-and-wayne-and-ian-and-donald-and-lynn-and-ralph
(snip)and-ralph-and-mike
(snip)and-ralph-and-mike-and-ed
(snip)and-ralph-and-mike-and-ed-and-chris
(snip)and-ralph-and-mike-and-ed-and-chris-and-simon
(snip)and-ralph-and-mike-and-ed-and-chris-and-simon-and-jochen
(snip)and-ralph-and-mike-and-ed-and-chris-and-simon-and-jochen-and...all-the-others

Whatever your reason for going, I look forward to seeing you next week. If you can't make it this year, hopefully we'll see you at EclipseCon 2011!

More bandwidth for eclipse.org

2010-03-18T05:45:00.000-04:00

Last Sunday we upgraded our saturated 80 Mbps Ethernet connection to the world with a shiny new Gigabit connection. Although it's capped at 100 Mbps this gives us an extra 20 megabits per second of throughput.

If you look at the graph above, you should know what times are best for downloading Eclipse. Above times are in the Eastern timezone.

Of course, when you get a Gigabit connection to the Internet, you just have to test it out... I lifted the bandwidth cap a couple of times to see how it would spike. In a matter of seconds we hit about 250 Mbps of throughput.

Strangely, the phone rang minutes later. Our ISP was calling, wondering if the massive spike was 'normal'. I guess they can't take a joke.

I'm not entirely happy with our rate limiting setup, nor with our QoS rules, but I'll be tweaking those in the weeks to come. In the meanwhile, enjoy the added bandwidth!

Quick poll -- what is your primary reason for going to EclipseCON?

2010-03-18T05:21:00.000-04:00

It's time for one of my whacky polls ... this time around, I'd like to know what is (are) your primary reason(s) for attending EclipseCON this year. You can vote in as many categories as you'd like but please -- don't kill my server :-)

http://eclipsecon.org/i-go-to-eclipsecon-for-the-free-beer

http://eclipsecon.org/i-go-to-eclipsecon-for-the-free-food

http://eclipsecon.org/i-go-to-eclipsecon-for-the-free-stuff

http://eclipsecon.org/i-go-to-eclipsecon-for-downloads--much-faster-than-download.eclipse.org

http://eclipsecon.org/i-go-to-eclipsecon-to-meet-up-with-friends-colleages-and-to-network

http://eclipsecon.org/i-go-to-eclipsecon-for-the-technical-stuff---talks-tutorials-bofs-its-all-good

http://eclipsecon.org/if-its-a-link-i-just-have-to-click-it--what-is-eclipsecon-and-how-can-i-download-it

I'll tally up the results sometime tomorrow. See you at ECon!

Hardware Upgrades Part 3: Solving problems 2, 3 and 5

2010-03-09T10:08:00.000-05:00

See Part 1: The Problems

See Part 2: Solving Problems 1 and 4

This is part 3 of 3 of the series, which describes how we'll deploy all the hardware donations we've received.

Problem 3 is the AMD servers that were donated at EclipseCon 2008. Great hardware all around, but we're running the Xen virtualization engine on top of it, and something obscure is preventing these machines from reaching their full potential. I could spend days, weeks and months trying to figure it out, but since they perform flawlessly when run on the bare iron, the solution here is to replace our AMD Virtual Servers (powering www.eclipse.org, Bugzilla, the Wiki, and others) with some new Intel hardware.

Problem 2 is the aging Itanium2 cluster that we know as dev.eclipse.org and download.eclipse.org. The AMD servers will be boosted with RAM and re-deployed here, as dev.eclipse.org and download.eclipse.org. With dual quad-core processors and gobs of RAM, we'll try to stay in RAM for your file access as much as possible.

Some Intel servers, boosted with RAM courtesy of Google, will be deployed here to help power git.eclipse.org.

Problem 5 is the lone Build server. IBM has just donated 2 awesome x3550 servers to help solve this problem, and we can re-deploy some Itanium hardware here too.

Since we'll have some leftover Itanium hardware, it will likely be re-deployed as backend server where appropriate, to take some load off NFS.

How's that for a plan?

IBM donates hardware for builds

2010-03-09T06:30:00.000-05:00

I am spoiled.

I have lost my right to complain about lack of hardware.

Rats.

After Google and Intel, it's IBM's turn to donate some hardware to the Eclipse Foundation. This time, a pair of IBM x3550 M2 servers, with two Intel Xeon 5504 Quad-core processors and a whopping 36 GB of RAM. Oh, and (3) 146G SAS drives attached to an LSI RAID controller.

This is the second time IBM has donated hardware to the Foundation. The first round included a trio of POWER5-based servers with a tray full of hard drives. Thanks, IBM!

Details on how these servers will be deployed will follow in the multi-part series I started here.

[gallery]

Have you seen the new Webtools website?

2010-03-04T06:00:00.000-05:00

Wow. It looks really polished, and I love the graphics treatment. http://www.eclipse.org/webtools/

Hardware Upgrades Part 2: Solving problems 1 and 4

2010-03-03T09:37:00.000-05:00

See also: Part 1

Solving Problem 1 is easy: just add bandwidth. With 10 new servers, we need more rack space, so I negotiated a good deal on some upgrades here as well. Now that our Cisco gear is multi-Gigabit capable, we can trade up that 100 megabit Fast Ethernet colocation port for a Gigabit port. We'll also add 20 megabits of permanent bandwidth, upping bandwidth from 80 megabits to 100 megabits.

Bonus side effects: packet latency will reduce with the increased signalling rate of Gig Ethernet.

Problem 4 is solved by a pair of Intel SR2625UR servers, boosted with more RAM thanks to Google. With 48G of 1066MHz RAM and eight Intel Xeon e5540 processors (sorry if this sounds like advertisement, but I love technical specs), these things will take complex SQL queries and spit out results in no time.

Other strategies include using a RAM disk for temp table creation. Four 10K rpm SAS drives attached to an LSI RAID controller means disk access will be fast, for those few times where disk is needed.

Bonus side effects: Since MySQL will be removed from the Power5 NFS servers, NFS will have much more memory for file cache. Also, reading and writing MySQL data, logs and temp tables will free disk seek time.

Hardware Upgrades Part 1: today's problems

2010-03-01T09:33:00.000-05:00

Since we've received hardware donations from Google and Intel, I figured I'd start a series on how we're planning on making Eclipse.org better. Let's have a look at the current problems we're facing.

Although there is room for improvement, it all works quite well.

Internet traffic is load-balanced across multiple servers for fault tolerance and scalability

Shared files are stored on a pair of NFS servers

Lots of content stored on local disks to avoid NFS

NFS servers also act as the MySQL servers

NFS/MySQL servers have three local storage devices: an 8-drive RAID, a 7-drive RAID and a 16-drive iSCSI RAID.

Since all our Cisco network gear was upgraded last year thanks for a very generous donation from Cisco, we're capable of sending multiple Gigabits/second to the Internet.

PROBLEM 1: Using 80% of a 100 megabit Ethernet cable doesn't give us much room to grow. I can only tap into another 12 megabits or so in case I need bandwidth fast.

PROBLEM 2: The 5-node dev/download cluster works well for redundancy and load capacity, but with only 8GB of RAM in each node, file cache hits are nil. Every request for a file, be it for CVS, a mailing list archive or a download, must come from NFS.

PROBLEM 3: Virtualized web servers make redundancy and scalability easy, but with only 8GB of RAM in the hosts, one host can only hold one instance of Bugzilla and www.eclipse.org. CPUs are largely idle. Furthermore, an obscure Xen issue is drastically affecting memory performance.

PROBLEM 4: Backend servers have 16GB RAM, which is OK, but must share that amount between MySQL and NFS. MYSQL is "detuned" to not consume too much RAM. Likewise, MySQL uses precious RAM that could be used for cache.

PROBLEM 5: Single IBM server for builds and signing. The machine is a monster, but four CPUs can only do so much at any given time. Also, Continuous Integration means CPUs are rarely idle.

PROBLEM 6: (not shown) Since we have a number of problems, adding new tools such as Git or Gerrit Code Review would only make matters worse.

In my next post, I'll discuss how we plan on addressing these issues with the new hardware we've received.

More hardware donations == happy webmaster

2010-03-01T04:36:00.000-05:00

After last week's donation from Google, this time it's Intel's turn to make the webmaster office a cheerful place.

I have just received the first of two shipments of donated servers: Eight (8) Intel SR1600UR servers, featuring two Quad-Core Intel Xeon E5540 processors, 12GB of RAM and a pair of 500GB SAS drives. I had specifically asked for very powerful, yet small 1U servers, and Intel sure has delivered.

This is Intel's second hardware donation to the Eclipse Foundation. Back in 2004, they donated the Itanium2 CPUs used in our current dev.eclipse.org/download.eclipse.org cluster.

Once loaded up with 24GB of RAM, these servers will provide extra power for our web servers (www.eclipse.org, Wiki, Bugzilla) and download.eclipse.org. In future posts I will go into more details on how both the Google and the Intel donations will make Eclipse.org better.

In the meanwhile, do I look happy or what?

Google is my new best friend

2010-02-25T08:12:00.000-05:00

I have just received a $20K USD donation from Google. Thanks, Google!

Some time ago, when bugs were opened asking for Git and Gerrit for Eclipse.org, I became grumpy; I whined and moaned that we didn't have the hardware resources to support that. It's not that Git and/or Gerrit needs tons of resources, but every additional service we support has a small incremental cost to the Foundation. Our servers are maxed out in just about every which way as it is, and adding more stuff certainly won't make it better.

But a guy by the name of Shawn Pearce heard my subtle, yet annoying cries for help and decided to put an end to my nonsense. After some discussions, he pulled some strings and managed to land a generous donation for Eclipse.org. The theme: make dev.eclipse.org better.

Stay tuned for details on how we'll upgrade our servers with this (and other) donations.

Git@Eclipse: let the pain begin

2010-02-01T12:19:00.000-05:00

Every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server.

-- Wikipedia

Did you know that agafgfafa @ yahoo dot com is also yours truly? Neither does Eclipse.org.

When you set up your workstation, the identity you give yourself is only as valid as you make it. Since anyone can commit to a git repository, including you@localhost, Denis@home and agafgfafa @ yahoo dot com, the commit log of any given repository is only valid if you can recognize the identity of the committers.

The problem arises when you push your commits to a 'central' git repository, such as what I'm setting up at Eclipse. Since git doesn't natively keep track of a push, the git commit log comes in to Eclipse servers unchallenged.

I'm sure you, nor the Eclipse IP team, would appreciate a commit log that looks like this:

commit 89aac775141bda1387429928d3601290f9959bf5
Author:     Denis <denis@localhost>
AuthorDate: Mon Feb 1 13:58:30 2010 -0500
Commit:     Denis <denis@localhost>
CommitDate: Mon Feb 1 13:58:30 2010 -0500

commit 3de863bdd0e5f579ff47438ed725184a6869f58e
Author:     Billy Bob <razorh4x0r@dsl163-1111.ny.nc.rr.com>
AuthorDate: Wed Jan 27 15:51:10 2010 -0500
Commit:     Big Daddy Denis <denis@fondation.eclipse.org>
CommitDate: Wed Jan 27 15:51:10 2010 -0500

commit 3027d0d773461d8cf2c96c4f13d754203629f00d
Author:     Denis <theman@homebox>
AuthorDate: Wed Jan 27 15:31:51 2010 -0500
Commit:     Denis Roy <denis@eclpise.org>

I've hacked an update hook (based on Shawn's clues) that will examine an incoming push and fail if it does not recognize all the committers, but it bears stating: committers will need to set this right before committing their code locally. Otherwise, the push will fail.

In the end, I'm sure it will all work out. It will just take a bit of getting used to.

Babel: no need to use map files anymore

2010-01-25T05:27:00.000-05:00

Participating in Babel, the crowd-sourced translations effort for Eclipse projects, just got easier: you will soon be able to point it to your update site or p2 repository, and Babel will figure out the rest for you.

Currently, Babel only supports parsing map files, then importing your English strings from your code repository. But since map files are not used by all projects, specifying your update/p2 site will make Babel easier to use.

For more info, see the mailing list post from our fearless leader, Kit Lo.

Bug 300,000 -- we have a winner

2010-01-18T15:26:00.000-05:00

Looks like Holger Voormann won my Bug 300K contest. Holger, I will coordinate with Lynn and get some Eclipse swag to you ASAP. Congrats!

I guess that means the next bug number of interest will be bug 314159. That could just take a few months at this rate!

Bug 300,000 -- who will have the honours?

2010-01-14T05:27:00.000-05:00

We're closing in on Bug 300,000!

mysql> select max(bug_id) from bugs.bugs;
+-------------+
| max(bug_id) |
+-------------+
|      299646 |
+-------------+

Let's have fun with this. If you open Bug 300K and it's a legitimate, real bug, contact me at webmaster@eclipse.org and I'll send you some Eclipse swag!

2009: from Bugzilla's perspective

2010-01-04T12:00:00.000-05:00

Bugzilla confirms it: 2009 was a great year for Eclipse, with 37,839 bugs opened and a whopping 53,466 bugs resolved.

Top bug reporters

480 Xiaodan Wang     @actuate.com
478 Markus Keller    @ibm.com
446 Tianli Zhang     @actuate.com
412 Dani Megert      @ibm.com
401 David Williams   @ibm.com
380 Paul Webster     @ibm.com
352 Andrew Overholt  @redhat.com
342 Steffen Pingel   @tasktop.com
335 Pascal Rapicault @ibm.com
324 John Arthorne    @ibm.com

Top bug resolvers

7851 Paul Slauenwhite @ibm.com
1140 Dani Megert      @ibm.com
1071 Steffen Pingel   @tasktop.com
1033 John Arthorne    @ibm.com
 710 Darin Wright     @ibm.com
 699 Eike Stepper     @esc-net.de
 627 Paul Webster     @ibm.com
 600 Curtis Windatt   @ibm.com
 534 Michael Rennie   @ibm.com
 531 Olivier Thomann  @ibm.com

Top bug commenters

11417 Paul Slauenwhite  @ibm.com
 5070 Denis Roy         @eclipse.org
 4836 Dani Megert       @ibm.com
 4367 Steffen Pingel    @tasktop.com
 3774 John Arthorne     @ibm.com
 3452 Susan F. McCourt  @ibm.com
 3081 Boris Bokowski    @ibm.com
 2861 Paul Webster      @ibm.com
 2702 Darin Wright      @ibm.com
 2630 Olivier Thomann   @ibm.com

I admit to being chatty, but I'm honestly surprised to make this list.

Most attachments submitted

682 Steffen Pingel      @tasktop.com
613 Xiaodan Wang        @actuate.com
548 David Green         @gmail.com
458 CDE Administration  @ibm.com
404 Matthew Hall        @woodcraftmill.com
400 Tomasz Zarna        @ibm.com
376 Kim Moir            @ibm.com
340 John Arthorne       @ibm.com
339 Maggie Shen         @actuate.com
319 Olivier Thomann     @ibm.com

Most contributed patches approved for IP Log

68 Ian Tewksbury       @ibm.com
51 Matthew Piggott     @piggot.ca
46 Ankur Sharma        @ibm.com
46 Chris Jaun          @ibm.com
44 Ian Bull            @eclipsesource.com
40 Tim Buschtoens      @eclipsesource.com
37 Pawel Pogorzelski   @ibm.com
36 Benjamin Cabé       @sierrawireless.com
31 Danny Ju            @oracle.com
29 Raksha Vasisht      @ibm.com

Do you have ideas for more queries or "top-X" lists that would be interesting? Post them in the comments.

99.973% uptime

2010-01-04T08:48:00.000-05:00

Uptime of our core services (Website, Bugzilla and CVS) has been on the rise, with an average availability of 99.973% for 2009. If there were 525,600 minutes in 2009, core Eclipse services were missing in action for about 142 minutes. Sadly, about half of that was because an ISP tech unplugged the wrong cable in May :( Oh well, accidents happen.

I'm hoping for 99.99% for 2010.

'tis the season... to donate to your favourite projects?

2009-12-15T06:11:00.000-05:00

I just donated some of my hard-earned money to two of my favourite projects:

- Wikipedia. I use Wikipedia several times a week when I want to read up on just about anything, so I usually make a donation every year. If Wikipedia ceased to exist, I would be very unhappy, so this is the least I can do.

- Amarok. With a good set of headphones on, amarok serves up the right mix of tunes needed to block out external distractions and allow me to be more productive. This is a sound investment (pun intended).

Of course, the Eclipse Foundation also accepts donations. If you feel you're getting great value from the best (and free) IDE/toolset available, a small donation to show support will go a long way.

Oh, that Bjorn!

2009-11-30T18:34:00.000-05:00

Great! Now that I have everyone's attention... :-)

You'll be thrilled to know that I just learned how to configure an NFS server. Yeppp. Folks have been doing this since 1973 and I just figured this out last week while a lot of you were eating turkey. Go try CVS. Or downloads. Or running a build. I bet you a beer at EclipseCon 2010 that it will be faster.

Bah, just try it. You know you'll get your beer anyway.

Hudson and Builds need hardware. But it's fall, and the money tree is bare. Every day, the UPS guy does not bring a large shipment of boxes in my name. But that's not stopping Dave Carver, Kim and I from trying stuff out. Stay tuned.

I'm adding e4 to git. Right now. You see, in this community, sometimes your bug is ASSIGNED, and soon RESOLVED FIXED before you even open it.

That is all. You may return to your regularly scheduled programming.

Friends paying friends for helping friends

2009-10-29T04:09:00.000-04:00

You don't need to pay your friendly Webmaster for saving your butt... But the funds will be put to good use (no, not drinks...) BTW, bug 292476 was initially called "Oops, I'm a dumbass!"

git repos at Eclipse!

2009-10-28T11:36:00.000-04:00

I'm in the process of setting up read-only git repositories of Eclipse CVS projects. You can see the list on this temporary page, and you should be able to use all the repos listed.

Feel free to read (and comment) on bug 280583.

An open SSH port means SSH attacks

2009-10-09T04:16:00.000-04:00

If your Internet-connected computer accepts SSH connections, chances are half the planet is trying to get in to hax0r you -- literally. Using Zeemaps and IP geolocation, we can easily plot the values on a Zeemaps.

I'm just kidding about Eric Rizzo.. But Eric, if that is you, don't count on me for free beer at EclipseCon :)

Nice project website

2009-10-08T05:46:00.000-04:00

I stumbled across the AMP project website this morning. The first thing that caught my eye was the terrific layout. The second thing I noticed was the context-sensitive top navigation that they are using. This is the first time I've seen a project override our $Menu object with some context-sensitive links. Cool! You can see how it is done by looking at the _projectCommon.php file.

Community Forums takes #3 spot for most popular URL

2009-10-05T06:19:00.000-04:00

The Community Forums homepage has taken the #3 position as the most visited URL at Eclipse.org, surpassing the main Downloads page. Here's the new short list:

1. "Find a mirror"

2. Eclipse.org Home

3. Community Forums

4. Main Downloads page

5. Resources

6. Categories

7. Documentation

8. EMF downloads

9. PDT downloads

10. Ganymede SR2 downloads

Five years @ Eclipse.org

2009-10-01T10:14:00.000-04:00

Today is October 1st -- my fifth anniversary working at Eclipse. To celebrate, I thought it would be appropriate to compile a THEN and NOW table of Eclipse.org, as seen by an IT guy. As the years go by, some things changed, and some stayed the same...

There is a picture here.

I'm sure that list doesn't even scratch the surface of everything we've done in the past five years, nor does it hint at all the awesome people I am privileged to work with on a daily basis. Here's to another 5 years of supporting this great community and helping projects pump out the coolest stuff!

Wow, what a painful release this was (is?)

2009-09-30T09:03:00.000-04:00

If you were able to upgrade to Galileo SR1/Eclipse 3.5.1 in a timely manner, you were probably just lucky. Even today, 5 full days after the release, our servers are still crawling.

What happened?

In late August, Karl and I had installed our new Cisco load balancer and firewall. Unbeknownst to us, we were dropping connections. A few committers noted that CVS connections were causing broken builds, and we had early reports from our mirrors that their RSYNC connections were being terminated. We didn't pay too much attention to the RSYNC issue in favour of resolving CVS, since RSYNC is one of those robust protocols that is essentially bomb-proof.

Mistake 1.

Fast forward to Friday, Sept. 25. I did real quick mirror check and everything checked out. We're good to go.

Mistake 2.

I mean, this is just a point release, and I've done millions of these. Business as usual, right?

Mistake 3.

At around 3:00pm ET on Friday, I was getting reports that the ZIP files were missing on most of the mirrors, despite the fact that they were considered in sync. Uh oh. Since Karl had found (and fixed) some short timeouts that may have caused the dropped connections, I went on to assume that mirrors were simply not yet fully up-to-date with Galileo SR1, and that they would be in sync sometime during the weekend.

Mistake 4

As it turns out, since late August, our mirrors would begin syncing, but would never finish. They were all badly out of date, but still considered in sync but because they checked in regularly. So they spent most of the weekend simply catching up, without actually getting the new SR1/3.5.1 files.

On Monday, the above became painfully apparent when we were caught serving p2 updates for most of the planet from a single 100 megabit internet connection. At this point, mirrors were having a difficult time pulling updates from us. I then brilliantly re-routed most of our downloads to our Amazon AWS account, after making sure it was in sync.

Wrong again, hero.

My uploads to AWS were also not completing. Apparently, when you update Eclipse, there are content/artifact jar files everywhere in our tree that need to be fetched. Some of those were not on AWS yet, causing the updates to fail.

"Epic Fail." What have you learned?

When you think it's business as usual, you're probably wrong. Plenty of learned lessons here.

What happens now?

Most of our mirrors are now in sync, and so is our Amazon AWS. p2 probably got burned by many broken mirrors and now only trusts the home site. It will eventually learn to trust its mirrors again. Until then, updates may be a bit slow, but they should succeed.

Galileo SR1 is here!

2009-09-25T07:26:00.000-04:00

Galileo SR1, based on Eclipse 3.5.1, is here! You can fetch your favourite goodies from the usual URL:

http://eclipse.org/downloads/

Galileo SR1 - available early for Friends of Eclipse

2009-09-24T06:44:00.000-04:00

Galileo SR1 is available a day early for Friends of Eclipse. Here is your link:

http://friends.eclipse.org/galileo_sr1.html

Of course, it's never a bad time to become a Friend:

http://www.eclipse.org/donate/

What is my NFS server doing?

2009-09-02T10:40:00.000-04:00

If you're running an NFS daemon (nfsd), at some point in time you may have wondered what it was doing right now. If it's running in kernel space, tools like lsof and strace don't work, so you're left guessing.

After much Googleing and some inspecting of the Kernel source code, I discovered some debugging values that can be poked into /proc/sys/sunrpc/nfsd_debug. The most useful was 32, which I used like this:

echo 32 > /proc/sys/sunrpc/nfsd_debug; tail -f /var/log/messages | grep lookup

Essentially, this will give you an idea as to what files are being served up by nfsd. Be careful, though: on a busy NFS server, this will spew lots of output to /var/log/messages.

After stopping the above command with CTRL+C, don't forget to turn off nfsd_debug:

echo 0 > /proc/sys/sunrpc/nfsd_debug

With this trick I was able to find some nasties that were hurting our NFS performance.

Fun times at work

2009-09-02T10:32:00.000-04:00

What a fun quarter this has been so far. It started with the new Forums site, then Matt and I performed some much-needed hardware maintenance, Karl and I swapped all our Cisco devices for new ones, and I upgraded Bugzilla last weekend. In the mix, I've been hunting down MySQL and NFS problems and looking for all kinds of optimizations to try to restore some snappiness to our site.

Both Bugzilla and the Forums still need a bit of work, and after that I tackle another big toy: Git.

Fun times!

Bugzilla 3.4 coming soon

2009-08-26T05:56:00.000-04:00

This Saturday I'll be upgrading our "ancient" Bugzilla 3.0 to the latest Bugzilla 3.4. Lots of new features will be available to everyone, and many of those are features our committers have requested. Just check out this dependency tree!

Here's a highlight of a few new features:

A new Guided Bug Entry wizard to help new users file bugs in the correct locations and reduce triage/duplication.

A much improved UI:

Uncluttered attachments by default:

.. and many more:

Better database storage for improved performance

Better email handling for improved performance

UTF-8 for better internationalization

I've set up a Bugzilla 3.4 sandbox for your testing enjoyment, and you can read about the new features in Bugzilla 3.2 and 3.4.

Squeezing more performance out of your Apache web server

2009-08-07T05:03:00.000-04:00

You can learn a lot about your web server by strace'ing it. Look at this, with my commentary in red:

www-vm1:~ # strace -p 5827
Process 5827 attached - interrupt to quit
poll(
  ^^Your shell will sit here until this process
    receives a request

[{fd=14, events=POLLIN, revents=POLLIN}], 1, 15000) = 1
read(14, "GET /modeling/images/dl-more.gif"..., 8000) = 677
  ^^ A request for a plain old GIF file

gettimeofday({1249652331, 82192}, NULL) = 0
stat("/home/local/data/httpd/www.eclipse.org/html/modeling/images/dl-more.gif", {st_mode=S_IFREG|0654, st_size=111, ...}) = 0
  ^^ This is the absolute path to the file on disk.

lstat("/home", {st_mode=S_IFDIR|0755, st_size=192, ...}) = 0
lstat("/home/local", {st_mode=S_IFDIR|0755, st_size=120, ...}) = 0
lstat("/home/local/data", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
lstat("/home/local/data/httpd", {st_mode=S_IFDIR|0755, st_size=112, ...}) = 0
lstat("/home/local/data/httpd/www.eclipse.org", {st_mode=S_IFDIR|0755, st_size=72, ...}) = 0
lstat("/home/local/data/httpd/www.eclipse.org/html", {st_mode=S_IFDIR|0750, st_size=4808, ...}) = 0
  ^^ Apache crawls the entire directory structure leading
     up to the file

open("/home/local/data/httpd/www.eclipse.org/html/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory)
lstat("/home/local/data/httpd/www.eclipse.org/html/modeling", {st_mode=S_IFDIR|0750, st_size=768, ...}) = 0
open("/home/local/data/httpd/www.eclipse.org/html/modeling/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory)
lstat("/home/local/data/httpd/www.eclipse.org/html/modeling/images", {st_mode=S_IFDIR|0755, st_size=2056, ...}) = 0
open("/home/local/data/httpd/www.eclipse.org/html/modeling/images/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory)
  ^^ Unless you have AllowOverride None, Apache will
     look for .htaccess files in each subdirectory of
     the DocumentRoot

lstat("/home/local/data/httpd/www.eclipse.org/html/modeling/images/dl-more.gif", {st_mode=S_IFREG|0654, st_size=111, ...}) = 0
open("/home/local/data/httpd/www.eclipse.org/html/modeling/images/dl-more.gif", O_RDONLY) = 15
close(15)                               = 0
read(14, 0x555555a356b8, 8000)          = -1 EAGAIN (Resource temporarily unavailable)
writev(14, [{"HTTP/1.1 304 Not Modified\r\nDate:"..., 170}], 1) = 170
  ^^ All of that hard work to simply respond "Use your cached copy"
     to the client!

write(10, "192.168.0.1 - - [07/Aug/2009:"..., 256) = 256
  ^^ IP address changed to protect the innocent.

poll(
  ^^ That request is complete, wait for another

All of this happens lightning fast. But do consider: Had the request been for a PHP file, all the above would have been repeated for each nested require(), require_once() and include() file. So the moral of this story is:

1. Don't nest your DocumentRoot too deeply. We could trim /home/local/data/httpd/www.eclipse.org/html to save lots of CPU and disk cycles

2. Keep AllowOverride None to avoid accessing .htaccess files everywhere, unless you really need them. We're investigating this seriously for www.eclipse.org.

3. Keep your web-visible directory structure short, too. http://www.eclipse.org/some/directory/structure/that/is/really/deep/and/nested/page.php will obviously generate lots of file stats (especially with AllowOverride)

4. If you use PHP files, only include what you need, otherwise you're stat'ing (and reading, and possibly compiling) PHP files for nothing.

The more you reduce disk and CPU cycles for one request, the more your web server will scale.

The endless list of Forums

2009-08-06T12:00:00.000-04:00

First off: I'd like to thank Eric Rizzo for helping out with the new Forums. As if he doesn't help out enough by answering questions and providing insight, now he's organizing the Forums, giving them descriptive names and pinging projects to get updated descriptions.

Now, on to the problem. There are about 115 forums for you to browse through, and as more and more projects get created, the page will only gather more clutter. Do you have any thoughts as to how all these forums should be organized? If so, we could really use your help in bug 284281.

99.999% uptime?

2009-08-04T09:04:00.000-04:00

Even with gazillions of dollars in the bank, it seems like the Five Nines eludes even Google.

Ready to kick the tires

2009-07-28T11:39:00.000-04:00

It's not done yet, but if you want, go ahead and try out the Eclipse Community Forums. The Control Panel needs work but the rest should function quite well.

If you see anything broken, please comment on bug 284281.

Newsgroups? No way, let's call them something else.

2009-07-22T09:52:00.000-04:00

While many[1] enjoy the finer charms of NNTP newsgroups, it's no secret that NNTP is one of those old-school protocols, predating the joys of Facebook, Tweetering and instant texting.

On some blog somewhere, someone suggested I look at FUDForum for Eclipse.org, since it's one of those cool Forum apps that has the ability to act as a front-end to nntp newsgroups. So I did, and I'm preparing to put this out for everyone to see. See bug 284281 for details.

But I have one problem: What do we call it? Eclipse Newsgroups? No way -- the cool kids will think we're stuck in 1971. Eclipse Forums? Eclipse Chat? Help!!!

Help me put a name to this page .. If I like your ideas, I'll send some Eclipse swag your way.

[1] Please tell me I'm not the only one...

Eclipse(.org) eclipsing Eclipses on Google

2009-07-16T06:06:00.000-04:00

If you do a Google search for Eclipse, you'll get something like this:

1. Eclipse.org

2. Eclipse Downloads

3. NASA

Youtube plug

4. Eclipse (software) @ Wikipedia

5. Eclipse Awards (nothing to do with us)

6. Eclipse (astronomy) @ Wikipedia

It's interesting that Eclipse (astronomy) is so overshadowed (pun) by our wonderful Eclipse OSS community. This has at least one person irritated (see yellow edit suggesting the addition of a disambiguation on the Wikipedia page for Eclipse (software)).

Maybe we should try to lower our Google ranking to avoid irritating other budding astronomers.

Well, this is embarrassing.

2009-07-13T09:25:00.000-04:00

Refreshingly honest. It makes the error all the more palatable.

Galileo should go down ...

2009-06-26T06:16:00.000-04:00

... as being the smoothest release I have ever participated in, despite the major constraint I had this year: no extra bandwidth shall be purchased.

Here is how we pulled this off:

1. More bandwidth. Since we got sooo many new Friends of Eclipse, I bent the rules and added 12 megabits of bandwidth [1]. But -- get this -- for the pre-release only. Marker (1) shows the increase, and marker (2) shows a significant drop three hours into the release. Unheard of!

2. Early access. Friends of Eclipse, mirror sites and Member companies all had early access to the bits, relieving some of the stress on release day.

3. BitTorrent. In April, I didn't have options (4) and (5) below, so with the no-extra-bandwidth rule, I had to set up something. Actually, I suspect many Friends took advantage of their benefits by downloading and pre-seeding many of the files before release day. More seeds == faster downloads.

4. Eclipse Member companies. They provided lots of bandwidth (and still are!) via the Get It Faster section of the download page. Many thanks!

5. Amazon Web Services. This was a last-minute deal that was/is a true blessing, since I was able to redirect lots of the download.eclipse.org requests to AWS instead of our stressed mirror sites.

6. Server virtualization. Our own version of 'the Cloud on demand' -- I enabled an extra virtual server node for www.eclipse.org on June 23 to handle the extra CPU power that would be required to handle the many requests. I may just shut it off later next week, or I may just keep it around...

I think we're getting good at these annual release trains..

[1] In reality, we get billed by the 95th percentile, so if I played my cards right, the extra bandwidth may end up costing us next to nothing.

Not cool

2009-06-23T10:51:00.000-04:00

This site makes it look like they're serving up Eclipse 3.5 already. Links all go to download.eclipse.org.

http://www.computerbase.de/downloads/software/entwicklung/eclipse_sdk/

Not cool.

Get ahead of the line for Galileo

2009-06-23T05:57:00.000-04:00

Galileo will be available tomorrow, but Friends of Eclipse and BitTorrent users can get it now. Beat the rush!

Eclipse IDE for PHP Developers

2009-06-19T04:50:00.000-04:00

A new IDE package will be avaible for Galileo this year: a PHP IDE. This package will include Eclipse 3.5, the PDT, Webtools and Mylyn.

I've been waiting a long time for this. This will make working fun again*. Get it on June 24, when Galileo is launched. Or, become a Friend of Eclipse and get it sooner.

* Note to my boss: working is still fun. I'm just adding drama for the blog post.

Design a Galileo Wallpaper

2009-06-09T03:59:00.000-04:00

If you have modest graphics design skills, you can likely create a Galileo Wallpaper in less than five minutes. If you attach it to this bug, then your name will be forever etched in the Eclipse Hall of Fame.

You can use some of Nathan's work on this page as a starting point. Perhaps Nathan can attach the EPS file(s) for it in that bug?

Babel language packs for Galileo need testing

2009-06-04T05:29:00.000-04:00

I have finally fixed most of our build process, and we're building language packs once again. Just download the Galileo language pack(s), unzip them into your dropins folder and launch Eclipse with the -nl (locale) switch. Report any problems against Babel.

We're not producing a p2 Update site for Galileo yet since we still have some issues to fix.

Galileo rush-hour traffic

2009-06-03T10:12:00.000-04:00

Can you spot the Friend of Eclipse?

Hey Denis, what is that on your screen?

2009-05-29T08:28:00.000-04:00

Oh, nothing of interest.

Optimizing SQL: maximizing query cache

2009-05-11T04:53:00.000-04:00

Today's topic is focused on getting the most out of MySQL's query cache. Query cache isn't on by default, so if your servers aren't using it, you're missing out.

Let's look at how the query cache works. Witness a query I've used before:

select translation_id from translations where string_id in
    (select string_id from strings where file_id = 35) and language_id = 2;
126 rows in set (3.08 sec)

If I run the exact same query again, the performance improvement is dramatic:

126 rows in set (0.00 sec)

So the query cache maintains a list of queries and their resultset, but add a simple space to the query, or capitalize the 's' in Select, and to MySQL, it's a whole new query. Furthermore, if the data is changed in any of the tables referenced by the query, the query and its resultset is cleared from cache and the query must be fully processed.

Don't be fooled by functions!

The following query, which returns changes that occurred yesterday and earlier, references the ever-changing value of NOW(), and will never be cached:

select translation_id from translations where string_id in
    (select string_id from strings where file_id = 35) and created_on < now();

Instead, use a static value if you can:

select translation_id from translations where string_id in
    (select string_id from strings where file_id = 35) and created_on < "2009-05-11";

Avoid costly updates

If your data changes so often that it's killing the effect of query cache, consider using separate tables for lookups (that get infrequent batched updates) and for updates.

As an alternative, if you're using a Master-Slave setup, configure your application to use the slave for SELECT queries and then configure low_priority_updates. A poor man's solution to preventing data from changing on your slave (to enhance query cache hits) is to cycle SLAVE STOP and SLAVE START commands with sleep intervals in between, so that data isn't constantly being written to.

At this stage, perhaps an even better (although more complex) solution would be to enable cache for the application to avoid touching the database altogether (memcached on PHP, squid for Apache, and so on.)

Of course, all of these solutions have drawbacks, so use them at your discretion.

Pre-cache large queries

The Dash project's Project Dashboards use a large MySQL database on a very small server. Once a week all kinds of information is gathered from CVS and SVN and all the tables are rebuilt. While once-a-week updates are great for query cache, the first hits to any given page can take a long time to run.

The inexpensive solution was to incorporate a series of wget commands into the script that rebuilds the data, so that the queries on some of the frequently accessed pages can be cached immediately after the build, thus rendering instantly for everyone else.

Alternatively, wget's output pages could simply be saved to static html (such as is the case for the Bugzilla homepage, with its top-10 queries) for even greater gains.

MySQL's query cache is a simple and effective way to squeeze performance out of your database. It is by no means the solution to all your problems, so use accordingly.

Up next: simple text search

Optimizing SQL Part 2: the size of your queries

2009-05-08T07:03:00.000-04:00

The size and complexity of a SQL query can greatly influence its execution time. Since MySQL 5.0, subselects are now supported, but I often see SQL novices overuse subselects with disastrous results.

For instance, consider these queries I use in Babel. For accurate results, I ran all the queries once, then cleared the query cache, then ran them again.

select string_id from strings where file_id = 35;
[snip]
83 rows in set (0.00 sec)

select translation_id from translations where string_id in 
    (select string_id from strings where file_id = 35) and language_id = 2;
[snip]
126 rows in set (8.18 sec)

If I replace the above subselect with the exact list of string_id values from the first query, look at the results:

select translation_id from translations where string_id in 
    (1072,1073,[snip: 80 others],197676) and language_id = 2;
[snip]
126 rows in set (4.94 sec)

In this case, without any further optimization, it's worth issuing two simple queries rather than a single subselect.

Of course, in() cannot compete with an inner join for this type of query:

select translation_id from translations 
    inner join strings on strings.string_id = translations.string_id 
    where translations.language_id = 2 and strings.file_id = 35;
[snip]
126 rows in set (0.02 sec)

Subselects are not always your friend. Using table aliases can then cut down on the amount of bytes you're sending to your server, reducing the load on your network stack, and making your query easier to read:

select translation_id from translations t 
    inner join strings s on s.string_id = t.string_id
    where t.language_id = 2 and s.file_id = 35;

Next tip: maximizing query cache

Optimizing SQL: Part 1 - minimizing connections and query counts

2009-05-07T05:22:00.000-04:00

www.eclipse.org is a pretty busy site, and for the most part, it's well optimized and quite fast. I'm not talking about Bugzilla or CVS here, only www.eclipse.org.

I'm a big fan of profiling and optimizing. Perhaps this is because I learned to program on a Timex Sinclair 1000 with 2KB of RAM, where Every Byte Counts!

In this multi-part series, I'll walk through some of the very simple optimizations I've done at Eclipse to make the site as fast as it can be with the hardware we have. The first installment:

Minimizing connections and query counts

You're likely very familiar with the Pick a Mirror page, the most accessed page on our site. That page needs database information to render, so I tend to keep it lightweight and optimized.

Since I recently added Bit Torrent as a download option, I wanted the Pick a Mirror page to show you the Torrent link if the selected file had one. I created a simple table called torrents_index which contained the name of the torrent file, so a simple pattern match between your file and the table would suffice.

But that involves querying the database. Our web servers and database servers are separate machines, so each query and response hits the LAN (generally).

Since the Pick a Mirror PHP script already queries the database for basic information about the file, I simply tapped into an existing query by adding a reference to TOR.torrentname (added content in red):

$sql = "SELECT DRP.drop_id, VAR.key_value, TOR.torrentname
	FROM drops AS DRP
	LEFT JOIN SYS_variables AS VAR ON VAR.key_name = 'download_table'
	LEFT JOIN torrents_index AS TOR ON (TOR.torrentname LIKE '%${filename_fileonly}.torrent')
	WHERE LEFT('$_file', LENGTH(DRP.our_path)) = DRP.our_path
		AND LOCATE(DRP.file_pattern, '$_file') > 0
	ORDER BY LENGTH(DRP.our_path) DESC LIMIT 1";
$rs = mysql_query($sql, $dbh);

In the above query, you'll notice that I'm fetching data from three different tables, DRP, VAR and TOR by using a JOIN, but the tables are not actually related in any way.

This simple technique of reusing a query is very efficient. It avoids another call to mysql_query(), and saves an entire round-trip to the database server. Overhead to the existing query is minimal to nil, as observed by timing its execution and showing the query plan with EXPLAIN.

Next up: Optimizing the size of your queries

Favorite Linux IDE

2009-05-05T06:27:00.000-04:00

My latest copy of the Linux Journal (yeah, an actual magazine, printed on paper... with ink), featuring the Readers' Choice Awards 2009, shows Eclipse as the winner of the Favorite Linux IDE award. This quote sums it up:

"In Eclipse, one can work in a lean environment and add and subtract an incredible array functionality with its myriad modules".

This quote was quite interesting:

"At the same time, the second largest vote-getter was “Other”.

Audacity, Amarok, OpenOffice.org, Adobe Air and, uh, Dell also got picked as Linux favorites. Ubuntu got the Distribution of Choice award, which is ironic for me, because I'm getting ready to ditch it and return to my Good Old Fedora.

Eclipse 3.5M7 torrents

2009-05-04T06:44:00.000-04:00

I've just put some Bit Torrents up for 3.5M7:

http://build.eclipse.org/technology/phoenix/torrents/SDK/stable/

Enjoy!

Where in the world is ...

2009-04-30T04:19:00.000-04:00

Have you ever wondered where all the Eclipse Committers and Contributors are? Have a look at the map above, and add yourself. It's Zeemaps, so no registration is needed.

Map of committers, contributors, users

2009-04-24T10:49:00.000-04:00

Matt took a few minutes to put together a map of all the Eclipse committers, contributors and users. Go add yourself -- you don't need to register or anything:

http://www.zeemaps.com/Eclipsecommittersworldwide

Let's fill this map up!

Eclipse torrents, part II

2009-04-23T11:36:00.000-04:00

In my never-ending quest to save everyone's bandwidth, I put together some bit torrent files for the EPP packages, and for the Eclipse Classic download. The tracker and seed are running on a minimalist virtual server, and I'm curious to see if it will be capable of handling the load that our marvellous Eclipse community can generate when a new Release is published.

If anyone wants to help me test this out, go ahead and seed your favourite downloads. If it works reliably, I'll kindly ask Nathan to add the proper links to the main downloads page.

Eclipse IDE for Java EE Developers (196 MB)

Eclipse IDE for Java Developers (85 MB)

Eclipse IDE for C/C++ Developers (67 MB)

Eclipse Modeling Tools (includes Incubating components) (310 MB)

Eclipse for RCP/Plug-in Developers (174 MB)

Eclipse IDE for Java and Report Developers (195 MB)

Eclipse Classic 3.4.2 (151 MB)

Help translate Eclipse in time for Galileo

2009-04-08T08:29:00.000-04:00

At Babel, the Eclipse globalization project, we're busy getting ready for the Galileo release. Our goal is to have the most complete, accurate and polished language packs ready for download with Galileo.

After fixing some stats issues, we were excited to see that Eclipse 3.4 is 98.8% complete for Chinese and French, and over 90% complete for four other languages. You can see all the Eclipse 3.4 stats here. Many thanks to Adobe for donating the full translation set for Eclipse 3.4 in six languages, and IBM's Eclipse 3.2 translations which were used as a starting point.

We're currently migrating the existing translations to the upcoming Eclipse 3.5/Galileo projects.

How can I help?

There are many ways you can help :

1. Get a Bugzilla account and start translating! You'll be productive in less than three minutes.

2. If you know PHP/MySQL, check out the Babel Server code and submit patches.

3. If you know Java, help advance the Messages Editor plugin.

4. Download and test the language packs we build.

5. Test Eclipse for proper string externalization with the Pseudo Language Packs. Report i18n issues with the projects.

git to replace CVS, Bugzilla, mailing lists, more at Eclipse.org

2009-04-01T09:18:00.000-04:00

With all the DVCS talk at EclipseCon and in The Bug, I quickly became convinced git is the way to go. With git, we can not only remove CVS and SVN, but Bugzilla also. Want to report a bug? Just file a git. Got a patch? Just commit it, and git will figure itself out.

Mailing lists? No more. Just send a git, and everyone will get it. Got it?

XMPP and IRC will also be deprecated -- you just need to git someone. For IRC channels, just set up a git hub.

Newsgroups and web forums are so passé. Just post a git.

Our MySQL databases will also be migrated to git. Instead of SQL queries, a git fetch will be used. Much more efficient, and there's no need for a central servers, as all fetches will be handled by peers.

These are exciting times!

The unsung heroes of EclipseCon

2009-03-30T11:09:00.000-04:00

We all heard about how Econ '09 was great, and how Bjorn is a superhero. But Bjorn cannot pull off such a great conference alone. I wish to highlight a few behind-the-scenes people that helped make '09 such a great conference. In no particular order:

- The Blue Shirt people. You probably didn't even notice them, but they keep the roof from falling.

- Scott Rosenbaum and the program committee. Your hard work shined in the richness of the program.

- Nathan & Ian. I mocked Ian's TwitterVision incessantly here in the office, but in the end, I admit it was really, really cool.

- The presenters and speakers. They didn't just read slides; they gave us hands-on, they walked amongst the audience, they made us laugh, think and learn.

What a great conference it was!

git off my lawn: DVCS discussions at EclipseCon

2009-03-30T10:47:00.000-04:00

There was a lot of discussion surrounding the DVCS topic at EclipseCon. Everything from Panels to Talks to BoFs to bar fights. Well, no bar fights, really.

In the end, at the second of two BoF sessions, small-ish group of people managed to come up with what looks like 'a plan'. Doug Gaff took some notes, including 'the plan', and I think he'll be posting those to the Wiki shortly. cc yourself to bug 257706 if you haven't done so already, and someone will post links to the Wiki minutes as they become available.

Person of the Week award

2009-03-25T11:40:00.000-04:00

There is definitely a buzz about EclipseCon 2009 so far. Folks are telling me the content is great, the organization is awesome and the ambiance is friendly. The one single "complaint" I've heard so far is that everyone hears about this Bjorn Freeman-Benson guy, but they don't know who he is.

If you see the man above in the halls of EclipseCon, give him a thumbs up!

CDO has a really nice project website

2009-03-24T14:20:00.000-04:00

http://www.eclipse.org/cdo/

Clean layout, nice icons, not too much text. Also, notice how the sub-pages re-use the icon from the main page for better association. On sub-pages, the left navigation is nicely done and the graphics headers are a nice touch.

Awesome work!