Friday, October 24, 2008

Build Workshop 3: Common Builder in its infancy

We're on Day 3 of the Build Workshop 3, and I'm seeing some really good stuff. Of course I don't understand most of it, but that's what I get from sitting between two rock stars like Nick Boldt and Andrew Overholt.

Folks from the Platform/PDE teams were here for Day 1 and Day 2 and provided tremendous insight on PDE, Ant and builds in general while Nick and Andrew hacked away on the very first iteration of the Common Build Infrastructure: a simple command-line shell script to launch a build. The system was proven to work as Nick managed to build GEF using it.

We've still got work to do before the Common Builder can be used by a large audience, but things are starting to shape up. You can have a look at the code so far. The Common Build Infra is hosted under the Dash project at Eclipse.org.

The team will be posting some notes, and likely a plan, on the Wiki page later on.

Wednesday, October 22, 2008

Eclipse Enterprise Users newsgroup

The Eclipse Enterprise Users group just opened today. Miles Daffin, who suggested the group, posted this welcome message:

"Are you responsible for providing eclipse at your firm? If so then this may be the group for you.

For the past 3 years I have been in charge of providing eclipse at my company. This presents some special challenges. The essence of the problem is that users are not allowed to download and install software from outside the corporate firewall. I therefore needed to create a private, secure replica of the outside world's Eclipse environment to service the needs of 1000+ java developers. In doing this I have encountered issues that do not seem to bother most other types of Eclipse user, so it is difficult to get useful responses in the other groups. There have been many times when it would have been really useful to discuss things with people 'in the same boat'."

Great idea, Miles!

Tuesday, October 21, 2008

Babel server now actively helps translate Eclipse

Thanks to a suggestion and a patch from Babel's newest committer Antoine Toulme, the Babel server now actively participates in the translation effort by finding translations for untranslated strings. There's a lot of string overlap across all projects, and on the French language, our beloved Eclipse Translation Genie has translated an additional 18% of the CDT 5.0 project, bringing its completion rate from 14% to 32% within an hour.

You can watch Genie's progress by refreshing the Recent Translations page. She has already finished scanning French, German, Spanish and Italian, and she's currently working on Japanese. Korean, Portuguese and Chinese will be next.

Genie's automatic translations are great, but she can't do them all. We still need your help translating Eclipse projects. All you need is an Eclipse Bugzilla account!

Monday, October 20, 2008

Build Workshop III: With a Purpose

Wednesday, Oct. 22 Nick Boldt and I are hosting our third ever Build Workshop. The goal is to implement a Common Build Infrastructure on Eclipse.org servers to allow Eclipse projects to focus on coding instead of release engineering tasks.

If you're a release engineer, or you know lots about building software, you're welcome to attend. It's in Ottawa, Ontario (Canada) this Wednesday, Thursday and Friday. Please read the Wiki page for all the details.

I'm really hoping this will be our last workshop on the subject, and that by Friday we have a Common Builder up and running -- even if it's a bit rough around the edges.

See you there!

Thursday, October 09, 2008

Linux servers: swap, memory and how much of each

When you set up a Linux server, you need to carefully consider the amount of RAM to buy and swap space to use. Back in the old days, folks would simply suggest setting the swap amount to twice the amount of RAM. Nowadays, as RAM is cheap, I see some people suggesting to not use swap at all. Here's my take on it. Let's use this top output from one node of our 3-node web cluster.



1. This box has 3.3G (let's just round the numbers for clarity) of available RAM. Let's find out if it is enough.

2. 2G of swap space is allocated, which is less than 1x RAM. As a general rule, I try to not allocate more than 2G of swap space for file servers *. I've seen (and still occasionally see) some of our servers that simply swap themselves to death (requiring a hard reset) as the disks become overwhelmed and the server unresponsive. The Kernel OOM killer does a fantastic job of keeping the box alive if it runs out of memory, so let it run out. If it's swapping constantly, there's a problem anyway, and more swap space isn't the solution.

3. About 2G RAM currently used. High RAM usage is not a bad thing, as the OS uses RAM for file cache for increased performance. Actually, unused RAM is wasted RAM in my book.

4. About 300M of SWAP currently used. Again, a high number is not a bad thing -- as long as it's not actively swapping (see below). Most decent OSes are smart enough to swap idle memory pages to disk if they're idle/sleeping for a certain period of time. This is a good thing, and a reason why some swap is good -- it frees sleeping RAM for active file cache duty.

5. About 880M of free RAM. In our case, we use the Apache prefork MPM, so some free memory is required to handle traffic spikes, where additional Apache processes would be spawned. However, I'd prefer this 880M go to the file cache -- if a traffic spike occurs, the memory manager will simply 'take back' memory from the file cache and allocate it to Apache. File cache memory is not committed (and lost) permanently.

6. About 74M of file buffers. These web servers don't write to disk much (except for Apache logs).

7. About 1.4G of file cache. Our entire web space is about 6G on disk, so it's caching a very large portion (25%) in RAM, which likely represents the most active web pages. In fact, if more web files were used actively, Linux would commit that 880M of free RAM to file cache, but in this case, it doesn't need to. The more active data you can get into file cache, the less often the server needs to go to disk.

8. 0.3% of CPU time is spent waiting for I/O (disk, network, etc). This is a good indicator that we're not actively swapping to disk, nor are we reading much web content from disk.


How do I know if my server is actively swapping?



Use the vmstat tool, and look at the si and so (swap in and swap out) numbers. vmstat 5 will report stats every 5 seconds until interrupted, so just sit there and watch it go. If si/so are greater than zero for extended periods of time, then your server is actively swaping pages of RAM to/from disk. In the very short 2-second sample above, we can see the following:

1. Despite having 302196 bytes in swap, 0 bytes are being traded between swap and RAM now. That means the swap contains inactive, sleeping data, and that's what we want it for.

2. bi and bo represent disk I/O: Blocks in (write to disk) and Blocks out (read from disk). Normally this is expressed in 1K increments, so you can see here that this server's disks are basically idle, reading 14K and writing 1K in the first sample, and zero disk activity 2 seconds later. Obviously, high si and so will also lead to high bi and bo.

Conclusion

Do set some swap space on your Linux Server (and Desktop), even if it's just 256M. Don't allocate 16G of swap for a basic file/web/mail server *, otherwise it may end up swapping itself to death. Let the OOM killer do its job, and add RAM if you're swapping to disk regularly. Large Application servers may benefit from more swap, as inactive/sleeping applications can be paged to disk to free up RAM for file cache or other applications.


* I consider an HTML/PHP webserver such as www.eclipse.org to be nothing more than a file server in the context of RAM/swap/disk I/O.


Edited: 4:33pm ET. Nathan pointed out a typo (who knew he could spell!)

Wednesday, October 01, 2008

Four years at Eclipse

On Oct. 1, 2004 I started my first day on the job with the Eclipse Foundation. If memory serves me correctly, that day -- my first day on the job -- I was in a meeting with the folks from IBM and the folks from our ISP (Primus) discussing

a) how we were going to move the IBM-hosted Eclipse server to our new ISP

b) what kind of resources we'd need (bandwidth, switching, routing and firewall gear)

c) plans to migrate all of Eclipse.org to a new set of servers

Talk about being thrown into action on the first day. Right then and there I knew I was in deep trouble it would be a challenging job, and it has been. It still is. And it's fun.

Just in time to celebrate my 4th birthday, Nathan has committed a brand new Eclipse.org homepage. Let's walk down memory lane and look at our website circa 2004-2005:

Home page, Oct 1 2004

Downloads Page, Oct 1 2004
Back then, if you wanted to download Eclipse, we sent you off to a mirror site right away. The mirror site may or may not work, and you had to decrypt wicked file naming conventions.

Community page, Dec. 2004

Downloads Page - first edition featuring the SDK and easy mirror selection, Apr. 2005

Look, no frames! Dec. 2005



Technology PMC is a bunch of badasses

I heard there was some movement in the Eclipse Technology PMC, so I wandered over to the Tech PMC page to discover the PMC is composed of some badass individuals.

Wayne "Mad Dog" Beaton. He's the lead, and therefore the most badass of them all.

Bjorn "Knuckles" Freeman-Benson

Chris "Iron Fist" Aniszczyk.

The only non-brute PMC members are Gunnar Wagenknecht and John Duimovich. But that Gunnar guy talks on the phone while being in the vicinity of what appears to be a beach, so he may have connections to "get the job done" without being anywhere in particular.

Excuse me while I go update my project plan, meta-data, ip log, re-read the dev process and perform an IP review of my own code.