Thursday, August 30, 2007

DLTK, Pokemon and distasteful pictures

We get lots of e-mail in the webmaster Inbox. Lots. We try to be as helpful as we can we each request. Being a relatively new Daddy, I couldn't help going beyond the call of duty by doing a bit of investigation for this particular e-mail (addressed to dltk-dev-owner, the DLTK developer mailing list administration address):


I can not find another contact or email for DLTK – perhaps you can forward to those who need to know. This morning my 7 year old and I were printing pokemon pages

So far, I'm thinking: Pokemon? What the heck are the DLTK guys doing???

and got quite a unexpected and shocking surprise of a topless lady in your advertising on the right hand side of the web page.

Advertising? Topless? Now I'm really wondering what these DLTK people are doing.

Please have this removed asap or I we will not be using your sites ever again!

I've heard this before, but not because of distasteful pictures on our site.

Here is the link (we were looking for Charizard pictures, under Fire pokemon).

(link removed, but it wasn't

Turns out it was one of those advertising glitches on a site linked with DLTK Kids, created by what appears to be a family of four whose initials are D, L, T and K. Poor parent.

Tuesday, August 28, 2007

RSS Feeds for Mailing Lists and Newsgroups

The community (that's you) have been asking us for RSS feeds for our mailing lists and newsgroups for awhile now (bug 107609, bug 106561, etc) and we've got them! It turns out that our mail archiver, Mhonarc, is a very versatile piece of software. It can take a template for it's output and can also handle more than one kind of index for each list. With a fair bit of work we can now supply RSS for any mailing list or newsgroup that is archived. Most public discussion is archived, so that means you can now subscribe to many lists via RSS if you don't want to participate, but want to read it. The one big exception to public list archiving are the CVS commits lists which are not archived for obvious reasons.

If you try to subscribe to your favorite list via RSS and find that it's not available that's probably for one reason: no one has posted to the list since August 18, when I first started generating the feeds. In order to produce the feed the archive needs to be updated and we haven't rebuilt the archives for all of the lists because of the disk intensity of this process (it has to process the whole list that way). So if you want a feed and the list doesn't have one, start a new discussion on the list. :) New messages are processed every few minutes.

So how do you find the RSS feeds? Each of the 'listinfo' pages will show whether or not the list has an active RSS feed as of 5am Eastern daily. If you go to the listinfo page like this one for wtp-dev:

You will see the new RSS icons. If you're in Firefox the feed will be displayed in the URL bar, and no matter what browser you're in it will be shown in the window as depicted here. If you've just sent a new message to the list the page won't reflect the new feed until 5am the next day, but it should be available within a few minutes. You can find it at[LISTNAME]/maillist.rss .

So enjoy!

Sunday, August 26, 2007

Eclipse: tools for IT people too

Although we have a policy of running OSS software exclusively here at, we also have a policy of not altering the source code of key software components. You know the drill: you hack into the code for all the right reasons, only to realize that months later, upgrading that hacked code to a newer version is a chore, if not impossible, depending on how much customization you've done. We do make small exceptions where it makes sense - such as the Bugzilla templates or, more recently, the Wordpress-mu code, which was altered to allow for authentication against the Bugzilla database.

Upgrading "hacked" code is pretty straightforward with Eclipse. The diff and merge tools (courtesy of the Platform Compare component) are second to none for this type of job. I start by loading the new, unmodified code in one Eclipse project and the copy of the hacked code into another project. I then use the Compare with Each Other utility, where Eclipse finds all the differences in all the files, also showing added or removed files. The compare tools then allow me to merge the changes one-by-one.

Merging code is a time consuming process, but Eclipse allows me to do it without losing my sanity.

Thursday, August 16, 2007

How do you email? The results are in.

Yesterday I conducted a poll because I needed your opinion on a few things. I just compiled the results:

145 keep-top-posting-its-fine
85 bottom-post-otherwise-youre-annoying
So I'll keep top-posting, except in mailing lists and newsgroups for the sake of keeping the thread sane.

87 your-404-page-is-ugly-make-a-new-one
I thought so. I opened bug 200281 for this.

116 plaintext-email-is-good
58 go-HTML-for-email
Plain Text it is! (I have a feeling someone did some ballot stuffing)

55 email-is-spelled-email-not-e-mail
Gotcha. I'll spell it email from now on.

45 how-can-you-possibly-live-without-facebook
Just watch me.

34 I-dont-read-your-boring-email-anyway
Yeah, well, I read yours... All of it ;-)

7 thats-me-denis-I-want-beer
Hmm, these mapped to an IP address in Germany. I wonder who that could be.

1 plain-text-default-HTML-if-replying-to-HTML
Good point, thanks.

1 new-school-old-school--dude-you-rock
Why, because I can't build a decent poll system? Thanks, though!

1 hey-Denis-hope-you-are-doing-well____signed_RealJenius
Yes, I am doing well, thanks for asking... I'm guessing you're a friend over at EclipseZone.

1 GET-is-not-the-safest-way-to-conduct-a-poll-think-pre-caching
Um... okay then!

Other interesting facts:
71% Firefox; 9% MSIE; 4% Safari
77% Windows; 12% Linux; 6% Mac

Now wasn't this fun... Thanks for your feedback!

Wednesday, August 15, 2007

How do you e-mail?

I'm old-school when it comes to the Internet communication. I spell e-mail with a hyphen. I top-post. I don't know what facebook is, nor do I really care to know. An emoticon is made up of ASCII characters, not graphics. I don't have a myspace page, and I still prefer Plain Text e-mails to HTML.

But tonight I received a copy of a reply message that I sent out on a mailing list. The previous e-mails in the thread were HTML, and my response e-mail PlainText'ed it all in an ugly mess, making me reconsider my old-school, low-frills preferences. What should I do? Please help me by casting your votes below:

Top or bottom posting:

HTML or PlainText e-mail:

Other recommendations:

I'll compile the results tomorrow night by grepping my Apache logs for /webmaster/.

Bug 200,000? That's worthy of some swag.

A few people have blogged about how we just passed bug 200,000. I hinted to Lynn that we should send the bug reporter some swag. Congrats, Benno, it's people like you who keep Bugzilla on its toes :)

Make money with Eclipse knowledge

I stumbled upon this juicy nugget (registration required) on the newcomers group this morning. Essentially, this gentleman is trying to do something in Eclipse and he's stuck, so he's offering $100 to the first person to help him.

I think this is the first time I see such a post. If I was in any way qualified to help him out, I would, and for free too -- just because I admire the fact that he understands the value of the time people put into these free-as-in-beer community support newsgroups.

Did I mention free beer?

Monday, August 13, 2007

A Groups Reorganization

We've grown a lot as a community over the past few years and as a reflection of that Denis has had to beef up the webmaster support staff. He can't fly as Lone Ranger any more and Matt and I are pretty darn busy, too. We had some discussions inside the Foundation about how to manage that and give us more time to give you guys better tools, better service, faster turnarounds, and at the same time keep growing the community--all without having to add more webmaster staff.

We decided we have some built-in inefficiencies we need to excise in order to be able to accomplish that. To get things rolling, Bjorn opened bug 198541 to talk about our strategy of managing Unix group to project mapping on So how do complicated group structures throw a wrench in the works?
  • Projects have widely differing group structures which makes operating on the CVS trees complicated to manage since every requested change has to be carefully cross-referenced before we can make it. This affects turnaround times. Bug 170829 is one example of what a mess this can be.
  • We have a situation where building automated tooling would be extremely difficult to implement. Tooling is what lets us spend less time on menial stuff and more time helping you with big issues. We would like to use the Portal to better advantage here but we are going to have a hard time leveraging it in the current morass.
  • We have a hard limit of 16 groups to which a person can belong because of NFSv3 (NFSv4 failed stability tests on our platform). Some users have to belong to more groups to do what they need to do. In order to work around this, POSIX ACLs have been set up to allow certain people to do their jobs. This affects turnaround times, and negatively affects projects as people who are in the ACLs accidentally assign group ownership incorrectly. Bug 157931 and Bug 126889 are examples of problems with the group number limit.
We want to create a real solution that works for you and works for us but the end goal is helping all of you get your work done faster, more efficiently, and more easily by letting us support you better.

Tuesday, August 07, 2007

It happens to them too!

My trusty left wingman Karl (my right wingman being Matt) blogged about a series of outages we've had this weekend due to hardware issues we've been having. I sighed in relief when, early tonight, I got this *twice* while trying to log into hotmail.

The error message was intermittent, meaning they likely have a few overloaded servers in the cluster.

So it happens to the best of 'em.

What the heck happened this weekend?

I can't post this in the most timely manner because in their infinite wisdom, Blogger have determined that eclipsewebmaster is a spam blog. Uh, what? Apparently too many links to one site. Gosh... it IS the Webmasters' blog. So I have put in a request to get this rectified and this will be posted as soon as they set themselves straight.

So what the heck happened this weekend? Some of you noticed that many services were not working properly over the weekend at certain points. This included: Bugzilla, the MyFoundation Portal, Eclipsepedia, and other services. Luckily CVS was not affected. So what happened?

We have two main back-end servers. These were kindly donated by IBM in 2004 and are quad Power5 boxes. One of them had a hardware/kernel interaction problem early last week that took down the site for a short time while we failed over to the secondary box. That machine then was shouldering the load for both servers. Everything was working fine for some time until on Friday night something locked up the database. Because we are running everything on these big boxes, we have one DB server for a lot of critical services. That's good because it's easy to build redundancy, but bad when the redundancy is out of commission. That's the situation we faced heading into the weekend. As soon as Matt brought the server back up it was slammed with a huge number of queued jobs, bringing it to a near standstill within 30 minutes. By early Saturday morning Pacific time, Matt and I had gotten the DB back on its feet and everything was humming along again. But we were still running on just the one server.

Early Monday morning one of the projects launched a very large number of queries in a very short amount of time. Each of these queries was running in its own DB connection. The result was that since we are running on a single server the total number of possible connections was eaten up by this job. Again everything came to a standstill. Denis took care of this and got the servers back on line. But by this point we had two fairly extended outages.

So that's where we stand. We're working hard to recover the downed server and bring it back online as a secondary. We are going to move some of the services around to make them more reliable. The root of it all is that we have reached the limit of the current architecture. Denis has us developing a plan to improve our hardware situation in next year's budget cycle and thus allow us to change the architecture. We're are doing our very best. In the meantime if anyone wants to gift us with a production-class SCSI RAID array and disks, or say a SAN array and cards, that might help us out now... ;)