Monday, March 22, 2010

EclipseCon: sometimes it's the small things

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

Sometimes it's the smallest things that can make a conference feel so much better.  The conference has only been "on" for two hours, and here's what I've noticed:

Breakfast: Fruit!  Plenty of fresh fruit.  A cereal bar.  Pastries.  But fruit!

Coffee: large cups that can hold more than 2 sips.  Folks can actually grab a cup and walk away from the fountain.

Centralized: Everything is contained within the hotel itself, so you don't spend half your day walking around looking for the rooms.  Much easier to meet up with friends and colleagues.

Props to the conference organizers who keep finding ways of improving an already great conference.

Friday, March 19, 2010

Got RAM?

NOTE: This is an old post that was picked up by PlanetEclipse as a result of moving my blog.

This is what 244 gigabytes of RAM looks like:

Thanks, Google!

Poll results: Why do you go to EclipseCON?

In yesterday's poll, I asked you what were your primary reason(s) for going to EclipseCon.  Here are the results:
41 /i-go-to-eclipsecon-to-meet-up-with-friends-colleages-and-to-network
39 /i-go-to-eclipsecon-for-the-free-beer
30 /i-go-to-eclipsecon-for-the-technical-stuff---talks-tutorials-bofs-its-all-good

It's a two-way tie for first place: you go for the great technical content and for the social/networking aspects of the conference.  I'm sure the high number of "free beer" responses are just a statistical error, nothing we need to pay attention to really.  Or, maybe it's a symbolic thing -- Eclipse is free as in beer, so you go for the free "beer".  Yeah, that's it.
24 /i-go-to-eclipsecon-for-the-free-food
23 /i-go-to-eclipsecon-for-the-free-stuff
20 /if-its-a-link-i-just-have-to-click-it--what-is-eclipsecon-and-how-can-i-download-it

I'll let you decide what to make of those numbers.
10 /

Well that's interesting.

Of course, others felt it appropriate to formulate their own poll responses:
1 /i-go-to-eclipsecon-for-the-free-beer-from-the-webmaster

Nonsense. I honestly don't know what that person is talking about.
1 /i-wish-I-could-go-but-boss^H^H^H^Hwife-won't-let-me
1 /i-dont-go-to-eclipsecon-but-i-really-wish-i-did

I can feel for these poor individuals that can not attend.  Maybe we'll see you next year?
1 /i-go-to-eclipsecon-to-meet-denis

I was really touched to read that one.  I really was.  Until I read the next responses:

Whatever your reason for going, I look forward to seeing you next week.  If you can't make it this year, hopefully we'll see you at EclipseCon 2011!

Thursday, March 18, 2010

More bandwidth for

Last Sunday we upgraded our saturated 80 Mbps Ethernet connection to the world with a shiny new Gigabit connection.  Although it's capped at 100 Mbps this gives us an extra 20 megabits per second of throughput.

If you look at the graph above, you should know what times are best for downloading Eclipse.  Above times are in the Eastern timezone.

Of course, when you get a Gigabit connection to the Internet, you just have to test it out...  I lifted the bandwidth cap a couple of times to see how it would spike.  In a matter of seconds we hit about 250 Mbps of throughput.

Strangely, the phone rang minutes later.  Our ISP was calling, wondering if the massive spike was 'normal'.  I guess they can't take a joke.

I'm not entirely happy with our rate limiting setup, nor with our QoS rules, but I'll be tweaking those in the weeks to come.  In the meanwhile, enjoy the added bandwidth!

Quick poll -- what is your primary reason for going to EclipseCON?

It's time for one of my whacky polls ...  this time around, I'd like to know what is (are) your primary reason(s) for attending EclipseCON this year.  You can vote in as many categories as you'd like but please -- don't kill my server  :-)

I'll tally up the results sometime tomorrow.  See you at ECon!

Tuesday, March 09, 2010

Hardware Upgrades Part 3: Solving problems 2, 3 and 5

See Part 1: The Problems

See Part 2: Solving Problems 1 and 4

This is part 3 of 3 of the series, which describes how we'll deploy all the hardware donations we've received.

Problem 3 is the AMD servers that were donated at EclipseCon 2008.  Great hardware all around, but we're running the Xen virtualization engine on top of it, and something obscure is preventing these machines from reaching their full potential.  I could spend days, weeks and months trying to figure it out, but since they perform flawlessly when run on the bare iron, the solution here is to replace our AMD Virtual Servers (powering, Bugzilla, the Wiki, and others) with some new Intel hardware.

Problem 2 is the aging Itanium2 cluster that we know as and  The AMD servers will be boosted with RAM and re-deployed here, as and  With dual quad-core processors and gobs of RAM, we'll try to stay in RAM for your file access as much as possible.

Some Intel servers, boosted with RAM courtesy of Google, will be deployed here to help power

Problem 5 is the lone Build server.  IBM has just donated 2 awesome x3550 servers to help solve this problem, and we can re-deploy some Itanium hardware here too.

Since we'll have some leftover Itanium hardware, it will likely be re-deployed as backend server where appropriate, to take some load off NFS.

How's that for a plan?

IBM donates hardware for builds

I am spoiled.

I have lost my right to complain about lack of hardware.


After Google and Intel, it's IBM's turn to donate some hardware to the Eclipse Foundation.  This time, a pair of IBM x3550 M2 servers, with two Intel Xeon 5504 Quad-core processors and a whopping 36 GB of RAM.  Oh, and (3) 146G SAS drives attached to an LSI RAID controller.

This is the second time IBM has donated hardware to the Foundation.  The first round included a trio of POWER5-based servers with a tray full of hard drives.  Thanks, IBM!

Details on how these servers will be deployed will follow in the multi-part series I started here.


Thursday, March 04, 2010

Have you seen the new Webtools website?

Wow.  It looks really polished, and I love the graphics treatment.

Wednesday, March 03, 2010

Hardware Upgrades Part 2: Solving problems 1 and 4

See also: Part 1

Solving Problem 1 is easy: just add bandwidth.  With 10 new servers, we need more rack space, so I negotiated a good deal on some upgrades here as well. Now that our Cisco gear is multi-Gigabit capable, we can trade up that 100 megabit Fast Ethernet colocation port for a Gigabit port.  We'll also add 20 megabits of permanent bandwidth, upping bandwidth from 80 megabits to 100 megabits.

Bonus side effects: packet latency will reduce with the increased signalling rate of Gig Ethernet.

Problem 4 is solved by a pair of Intel SR2625UR servers, boosted with more RAM thanks to Google.  With 48G of 1066MHz RAM and eight Intel Xeon e5540 processors (sorry if this sounds like advertisement, but I love technical specs), these things will take complex SQL queries and spit out results in no time.

Other strategies include using a RAM disk for temp table creation.  Four 10K rpm SAS drives attached to an LSI RAID controller means disk access will be fast, for those few times where disk is needed.

Bonus side effects: Since MySQL will be removed from the Power5 NFS servers, NFS will have much more memory for file cache.  Also, reading and writing MySQL data, logs and temp tables will free disk seek time.

Monday, March 01, 2010

Hardware Upgrades Part 1: today's problems

Since we've received hardware donations from Google and Intel, I figured I'd start a series on how we're planning on making better.  Let's have a look at the current problems we're facing.

Although there is room for improvement, it all works quite well.

  • Internet traffic is load-balanced across multiple servers for fault tolerance and scalability

  • Shared files are stored on a pair of NFS servers

  • Lots of content stored on local disks to avoid NFS

  • NFS servers also act as the MySQL servers

  • NFS/MySQL servers have three local storage devices: an 8-drive RAID, a 7-drive RAID and a 16-drive iSCSI RAID.

Since all our Cisco network gear was upgraded last year thanks for a very generous donation from Cisco, we're capable of sending multiple Gigabits/second to the Internet.

PROBLEM 1: Using 80% of a 100 megabit Ethernet cable doesn't give us much room to grow.  I can only tap into another 12 megabits or so in case I need bandwidth fast.

PROBLEM 2: The 5-node dev/download cluster works well for redundancy and load capacity, but with only 8GB of RAM in each node, file cache hits are nil.  Every request for a file, be it for CVS, a mailing list archive or a download, must come from NFS.

PROBLEM 3: Virtualized web servers make redundancy and scalability easy, but with only 8GB of RAM in the hosts, one host can only hold one instance of Bugzilla and  CPUs are largely idle.  Furthermore, an obscure Xen issue is drastically affecting memory performance.

PROBLEM 4: Backend servers have 16GB RAM, which is OK, but must share that amount between MySQL and NFS.  MYSQL is "detuned" to not consume too much RAM.  Likewise, MySQL uses precious RAM that could be used for cache.

PROBLEM 5: Single IBM server for builds and signing.  The machine is a monster, but four CPUs can only do so much at any given time.  Also, Continuous Integration means CPUs are rarely idle.

PROBLEM 6: (not shown) Since we have a number of problems, adding new tools such as Git or Gerrit Code Review would only make matters worse.

In my next post, I'll discuss how we plan on addressing these issues with the new hardware we've received.

More hardware donations == happy webmaster

After last week's donation from Google, this time it's Intel's turn to make the webmaster office a cheerful place.

I have just received the first of two shipments of donated servers: Eight (8) Intel SR1600UR servers, featuring two Quad-Core Intel Xeon E5540 processors, 12GB of RAM and a pair of 500GB SAS drives.  I had specifically asked for very powerful, yet small 1U servers, and Intel sure has delivered.

This is Intel's second hardware donation to the Eclipse Foundation.  Back in 2004, they donated the Itanium2 CPUs used in our current cluster.

Once loaded up with 24GB of RAM, these servers will provide extra power for our web servers (, Wiki, Bugzilla) and  In future posts I will go into more details on how both the Google and the Intel donations will make better.

In the meanwhile, do I look happy or what?