Tuesday, June 26, 2012

Eclipse Juno is here now for Friends!

Eclipse Juno, the 2012 Eclipse release based on the Eclipse 4.2 platform, will be available tomorrow, June 27.  But it is available NOW to Friends of Eclipse.  Become a Friend and Download it Now.

Monday, June 04, 2012

Problems of scale

Recently I noticed that the web logs of the Eclipse Download servers were being overrun by an inordinate amount of requests for various p2.index files.  All the responses appeared to be "404 Not Found".  This seemed odd to me, so I began to investigate.

My findings: 6,000,000 each day. We respond to 6 million requests for p2.index files each day.  Ouch.

Why not?

I won't go into the "why?" but I do want to elaborate on the "why not."  "Checking for a small file is a trivial request, no one will ever notice, impact will be minimal" is usually the thinking.  And that thinking is correct -- go ahead and click this link, and it will all be over before you know it.

Small, insignificant request

Problems arise when small, insignificant requests add up by the millions.  An Ethernet packet cannot be smaller than 64 bytes, so although responding to a "404 Not Found" is only a few bytes, this adds up to 384 MB of actual data sent on the wire each day.

Where have all the bandwidths gone?

Unfortunately, there is more.  Until recently, Eclipse wasn't able to reuse an http connection for multiple requests, so it would go through a 10-packet TCP exchange for each and every file it wanted.  That's 4 Ethernet packets departing the server for that p2.index "404" response.  All of a sudden we're sending 1.5 GB of 404 material.

It's happened before!

Client-server developers typically don't know much of what's going on server-side, there's nothing new there.  I remember many years ago when the Mylyn (née Mylar) project was killing Eclipse's Bugzilla by fetching too many bugs and too many repository listings too often when their project became wildly successful.

But here's one small tip: if you catch yourself saying, "it's small and insignificant", "no one will notice" or "no impact", go ahead and multiply the action by 6 million to see of the answer is the same.

Or engage with your IT team.  They want to help  :-)