Wednesday, July 08, 2015

Diversity from an automotive racing perspective

Diversity in tech is an issue that needs no introduction. Although, I just did that.

I attended a large professional automotive drag racing event last weekend, and coming from a tech background, the diversity in that field was so obvious that it just goes unnoticed to everyone else. The leaderboards are peppered with women and men. The biggest names in racing are women and men. The crowd cheers for women and men. The attendance itself is women and men.

And children. Young children. Teens. Boys and Girls.

In fact, children are often seen in the racing pits, watching helping Daddy and Mommy repair an engine, prepare the car for the next round and do whatever it takes to win.

Where are the children in tech? Where are the children at EclipseCon? Where are the children in your office building?

My daughter in 2010, at age 4, watching a race from afar while I change tires.

We involve our children into our leisure activities at a young age -- be it art, sports, or automotive racing. But I don't see the same type of involvement in tech.

My expertise on this matter is only as exhaustive as my observation skills, but here is what I am doing to encourage my daughter to go into tech:

- I bring my daughter to the office for 1/2 days when I can. She spends most of her time doing activities she prefers, and I occasionally show her what I'm working on and why it's important.

- I bring my daughter to the data center, where the Eclipse servers are running. She is not allowed to touch, and I only spend a few minutes there since it's quite cold.

- I show my daughter pictures and content of the tech conferences I go to.

- I relate my daughter's homework to what I do for a living.

... and I will do all the same with my son, as soon as he reaches an appropriate age.

If we want to improve diversity in tech, I think it has to happen at an early age. If Demo Camps, hackathons, conferences and even daily office activities were more children/family-friendly, perhaps diversity in tech would be like diversity in professional drag racing:

A non-issue.

Friday, February 06, 2015

Using Bugzilla with a phone -- no apps!

Lately I've had to post comments on bugs but all I had was my aging Android phone with me.

Painful.  Just painful. Bugzilla's UI is dated, and that becomes even more apparent on a device.

Get used to zooming

There's a Bugzilla App, and I'm sure it's good, but I don't want to do Bugzilla on my phone.  I just need the UI to tolerate small screens long enough to do a few common tasks.

So I found Bugzilla bug 101865: support for small devices, opened in September 2001. Yeah, that's not very encouraging. But after some exchanges with the Bugzilla team, I felt encouraged and was motivated and decided that a prototype was in order.

Edouard jumped at the challenge and had Bugzilla running in Docker in no time.  After a few HTML tweaks and many (many) CSS magic tricks, he came up with a nice, as-responsive-as-it-can-get interface that actually works quite well on a phone.

Edouard's DuskMobile theme makes Bugzilla look clean.

These tweaks make it MUCH easier to log in, search, browse and comment on bugs using a mobile device. No app required.

We know that mature projects (like Bugzilla, and Eclipse) don't like big chunks of revolutionary change all at once, so we tried to keep the changes small and contained. Bootstrap? Not yet.  Small steps.

Nice work, Edouard. I'd be curious to see what you could do if we could change the nested tables to DIVs and could use Bootstrap.

Friday, January 24, 2014

Results: EclipseCon 2014 is returning to California... What are your thoughts?

The results are in!!  Earlier this week I posted a whacky poll, asking you what your thoughts were on EclipseCon's return to the Bay area.  Here are the results:

36 Im-glad-its-back-in-SF-I-hope-the-weather-is-nice

32 Im-still-hopeful-for-an-EclipseCon-in-Maui-make-it-happen

30 I-prefer-the-east-coast-less-time-on-a-plane

No surprise there.  I enjoyed driving to the last two EclipseCons but I'm looking forward to sunny California!

16 doesnt-matter-where-ECon-is-I-will-be-there

11 Im-glad-its-back-in-SF-I-will-visit-with-family-and-friends

6 what-no-beer

I have to keep my material fresh, so there were no choices for beer.  Sorry.  But that didn't stop some poll veterans from making up some choices.



Looks like I owe some kind folks a beer or two.  Less than 2 months to EclipseCon North America, see you there!

Wednesday, January 22, 2014

Poll: EclipseCon 2014 is returning to California... What are your thoughts?

There was a time when EclipseCon was as West Coast USA as the Golden Gate bridge.  However, unlike the bridge, EclipseCon has been to Washington, D.C. and to Boston, Massachusetts to broaden its horizons.  Next March, the favourite gathering point for the Eclipse community is returning to the bay area for its 10th anniversary.

So here's the poll: What do you think about going back to San Francisco?  Clickety-click the links below to cast your votes:

Next week I'll post the results!  Remember, no ballot stuffing or reverse-engineering of the polling process!

Friday, November 29, 2013

Using Gerrit code review without code review

Committers who typically push to HEAD (the master branch) can hesitate to enable Gerrit simply because they don't want to add the extra step of reviewing code. Fortunately, with Gerrit, you don't have to.

In fact, EGit makes it easy to push directly to master on a regular basis yet still submit a change to Gerrit for code review.  Here's how:

1. Ask Webmaster to enable Gerrit on your repo, and explicitly state that your project committers should be able to bypass code review.  Moving forward, Gerrit is the only system that can write to your Git repo, even if you're pushing directly to master.

2. Using EGit, clone your repo using the Gerrit URLs, not the Git URLs.  These typically look like this:

I like to use the "Import from Git repository" functionality in Eclipse, since it will clone the repository and create a project at the same time.

3. By default, EGit will be configured to push directly to the master branch.  I use the Commit and Push button to bypass code review, and the Commit button to (later) push the changes to Gerrit for review.

EGit "Commit" dialog.

If Webmaster didn't activate the Push to Master permission, you'll likely see this:

Gerrit prohibits direct push to master

4. You're done.  Use the Commit and Push button to push directly to the master branch.  If you do wish to create a code review for your local commits, choose the Push to Gerrit item:

Right-click > Team shows Push to Upstream vs. Push to Gerrit

Push to Gerrit allows you to specify code review branch... in this case, refs/for/master, which differs from the master branch, refs/heads/master

5. If Push to Gerrit succeeds, you've effectively created a change, and Gerrit will provide a URL:

6. In the Gerrit UI, you can review your change then Submit it. Gerrit then merges your review branch into head and your change status becomes Merged.  Or, you can request review from others, who can  then Publish review commentary or Publish and Submit 

Gerrit code review

7. Watch your new Gerrit project!  Contributors may be submitting changes to Gerrit right now, and they are awaiting your review.

Monday, November 04, 2013

HIPP: Making Hudson work at

At Eclipse, we've been using the Hudson CI server since 2008 when it was originally set up by members of the Eclipse community -- a handful of committers who wanted to take advantage of a great CI system and put their release engineering online, available to all.

After a while, the service was deemed important and useful, and management responsibility was transferred to the Webmaster team.  Projects signed up, the number of servers grew, and the variety of plugins increased as build methodologies and complexity varied from project to project.  Meanwhile, performance and stability were not usually on our side, and we struggled to keep the service problem-free for any amount of time.

If you build it, they will come (and destroy it all)

To be fair, any piece of code will get a stress-test when exposed to a large, open community such as Eclipse, where bots, scripts, users and search engines will use and abuse every nook and cranny, 24/7.  Nifty features, such as viewing logs or downloading workspaces as a ZIP file work great on internal or small-scale CI systems, but for a master with eight slaves, hundreds of jobs and thousands of users, it can be a bit overwhelming.

The Hudson project team has always been very responsive to our requests.  They've examined stack traces, bugs and build jobs.  However, in the end, problems typically revolved around any of the following states:
  • Hudson isn't designed to be a web server, responding to tons of http requests.
  • Hudson isn't designed to be a file server, serving 100+ MB files over the network.
  • Hudson isn't designed to be a log viewer, where some build logs can get quite large.
  • The design behind the slave delegation isn't great.
  • None of this is easy to fix.
As my Java Developer days are but a faint memory, picking up the codebase and hacking out patches was likely only going to make things worse.  Instead, I thought -- why not create a Hudson setup where it actually shines: in a smaller, more focused build environment. Enter bug 403843: Hudson Instance Per Project, or HIPP as I named the idea of providing Eclipse projects with their own dedicated Hudson instance.

Old school solutions to new problems.

Having dozens of Hudson processes on a single server, all performing specific tasks for individual projects didn't seem as scary as having a single Hudson process performing tasks for dozens of projects.  After all, threading, concurrency and resource contention are all issues that are resolved at the Operating System level.  The Linux Kernel will always be better at load-balancing multiple tasks than Hudson (and Jenkins) will ever be.

The Webmaster team got to work setting up the first HIPP server, and by leveraging Puppet we ensured we could easily create dozens more just like it.  By late July 2013, the first HIPP instance was set up as a proof-of-concept for the Sapphire project, and a quick "ps aux" command shows me the same Hudson instance is still online and operational since August 9.  So far, so good on the stability front.

RAM -- for breakfast, lunch AND dinner.

Of course, the proposed solution had some potentially serious hardware implications.  An idle Hudson master can take about 600M of RAM, and 4 to 8GB (and more) while running a build.  Banking on the fact that not every project builds at the same time, we went the conservative route and decided to allocate about 12 HIPP instances to a single 64GB server (24 instances to a 128 GB box).  Casual observations lead me to believe we could easily double those numbers without oversubscribing the box, but we're going into this cautiously. Stability first! We'll readjust our targets as we gain experience with the new setup.

Hudson -- he's a member of your project now.

Get to know the Hudson Butler -- that gray-haired, mustache-adorning gentleman wearing a suit and red bow tie, because under HIPP, he can be a committer in your project group, with the same rights to tag and branch your Git repo, as well as drop signed files directly into your downloads area.  A large, monolithic Hudson serving multiple projects can't do such things, since I've always had reservations about storing committer credentials inside a web app, or allowing one app to delete files in anyone's project.

Since a HIPP is now working specifically on your project, we've also allowed a host of plugins on HIPP that are still forbidden on the "shared" Hudson, such as the Gerrit plugin, which allows Hudson to be an active participant in your code review process.

What about Hudson Team Support?

Before diving into HIPP, I had some very interesting discussions at EclipseCon with some of the Hudson project members, and that's where I was introduced to the soon-to-be-released multi-tenant aspect of Hudson 3.1.  The promises were, well, promising, but it was not released yet, and I needed to fix stability years ago. I couldn't wait. Besides, I wasn't entirely sure that the Hudson team support could give me the levels of user/project separation I was after, so I forged ahead with HIPP.

As I write this, Thanh is provisioning HIPP instance #37, I'm working on web-based HIPP self-serve start/stop/restart, we're getting positive feedback that it all to Just Works and that makes me a happy camper.

Keep on building.

Wednesday, March 20, 2013

Poll results: How do you prepare your EclipseCon talks

The results are in! In yesterday's poll I asked how you prepared for your talk(s) at EclipseCon. Here were your answers:

30 i-prepare-my-talks-on-the-plane-to-econ

27 oh-crap-ill-be-back-later-theres-something-i-must-go-do

22 i-prepared-my-talk-in-2009-maybe-i-should-update-it

19 i-prepare-my-talks-while-i-drive-to-econ

16 i-prepare-my-talk-when-i-get-there

124 I-assumed-Denis-was-preparing-my-talks-for-me

If you think I am in some way qualified to prepare your talks, the attendees are in for a painful experience.

56 like-Eric-I-prepare-mine-on-the-toilet

That is way too much information.  I'd be leery of printed handouts at those talks.

For some reason, there were more answers on the topic of beer than any other.   I am shocked!


Congratulations -- you are in for a great week. Catch me at the bar and I'll buy you a beer.  You'll need to be fast since I am not at the bar very often.


In this age of Drupal-enabled websites, this is out of my control.



I am sorry.  I will try harder this year.

5 /

I know who you are.

5 /

I will run with you -- to the beer store!


Thanks to everyone who participated!  I look forward to seeing everyone again at EclipseCon 2013!

Tuesday, March 19, 2013

EclipseCon Poll: How are you preparing your talk(s) for EclipseCon

It's time for another Webmaster Whacky poll, where failure is the only sign of success.  In less than a week we'll all be at our favourite hangout, EclipseCon 2013.  This time I ask a simple question:

How do you prepare for your talk(s) at EclipseCon?  Cast your votes by clicking the links below:

I'll tally up the results tomorrow afternoon.  Remember -- no ballot stuffing!!

Wednesday, February 27, 2013

Cisco CSS: Load balancing from the inside too

Disclaimer: I'm not a Cisco expert.  Years ago, then-webmaster Karl Matthias convinced me that I was almost smart enough to barely understand this gear.  Turns out he was right.

The skinny

We use a Cisco CSS to load-balance client requests to multiple servers.  For years we couldn't load-balance requests from our inside network, only from the outside.

The setup

If you read the Cisco docs, the predominant use-case for a load balancer appears to be a single CSS with  a  single server group serving all the content. However, like at most shops, we have multiple server groups to serve different content.  For example, we have three servers to serve, two for Bugzilla, three for Git, three for, and so on.

Here, the Load Balancer acts as the gateway -- all inside servers use rfc 1918 private IPs and use as their default gateway.  One /24 subnet is used: 172.16.0.X. The CSS has multiple "virtual" IPs: the real, Internet-routable IP addresses that represent the services.

For Internet clients, this setup works beautifully.  When you consider that a CSS is nothing more than a heavy Spoofing device, you can easily follow the flow of traffic from a client, say

  • Client sends SYN packet to, which is
  • The CSS immediately responds with a SYN-ACK, which is ACK'ed by the client, thus completing the three-way handshake
  • Meanwhile, the CSS spoofs the connection to one of the real servers in the group.  It crafts a new SYN packet --  Source:  Dest: (it happened to pick that one)
  • The real server responds with a SYN-ACK: Source: Dest:  Since the Destination is remote, the packet is sent to the Default Gateway, which is the CSS (
  • The CSS simply discards the SYN-ACK since it has already established a socket with the real client. It ACKs the real server and completes the three-way handshake on the backend.
  • Everyone is happy, and traffic is free to flow from the client to the real server.

The problem

Problems arise when a server on the "inside" becomes a client to a load-balanced service, also on the inside. For some reason, it just doesn't work.  Years ago, the Cisco experts (not Karl) just told me it was how Cisco devices worked -- the load balancer is not meant to be accessed from the inside network.  The Cisco forums provided no particular guidance, other than to essentially "NAT" the inside servers as clients.  While that solution works, I didn't find it particularly pretty.

We originally resolved the issue with hosts file entries at first, then internal DNS.  Since all our servers share a backend network connection, server-to-server connections would flow over it.  It worked, but it was error prone and confusing.  If one load-balanced node died or was taken offline, we'd need to remember to update DNS.

Why it doesn't work

The years passed and I didn't spend much time thinking about it, but as our services grew in number, size and traffic volume, the problems became more frequent and annoying.

Understanding the root cause of the problem was key to developing a solution, which happened haphazardly while explaining it to a bunch of Linux students.  A light bulb lit up and I saw the light.

The following day I spent a bit of time with tcpdump and webalizer, I noticed that the internal "client" trying to reach an internal server from the CSS was eventually receiving two SYN-ACK packets.  The client, understandably confused, would RST the connection leading to failure.  Bingo.

Following the flow of traffic from the inside "client", the problem becomes apparent.  Say Bugzilla server wants to talk to server, using the virtual IP:

  • Internal Bugzilla server (the client) sends SYN packet to, which, through the magic of DNS, resolves to Remember, that IP is the CSS.
  • The CSS immediately responds with a SYN-ACK, which is ACK'ed by the bugzilla server (the client), thus completing the three-way handshake.  So far, so good.
  • Meanwhile, the CSS spoofs the connection to one of the real servers in the group.  It crafts a new SYN packet --  Source: (the bugzilla "client")  Dest: (one of the www nodes)
  • The real server responds with a SYN-ACK: Source: Dest:  Are you seeing this?  Unlike earlier, this time the Destination IP is not remote, the packet is not sent to the Default Gateway.  The SYN-ACK is sent directly to the Destination.
  • Two things happen:  1) The "client" receives a second SYN-ACK -- one from CSS, which is the spoofed connection, and now one from the real server.   2) The CSS is not "seeing" the response.  The CSS must "see" all the traffic.
  • Bugzilla server (the client), confused by the two SYN-ACKs, issues a connection RST and the connection fails.

The solution

For internal load-balancing to work, the CSS must see all the traffic coming in-and-out.  The easiest solution here is to isolate the servers in content groups to their own subnet.  Consider this:

The changes may be hard to spot:

  1. No changes to the "www" servers.  They remain on the 172.16.0 subnet, with a 24-bit mask.
  2. Bugzilla server IP addresses change from 172.16.0.X to 172.16.1.X.  Also with a 24-bit subnet mask, they are now on a different IP subnet than the www servers.  Physically, no wiring or vlan changes are needed.  Default Gateway changes to
  3. On the CSS, a new IP address is assigned to the inside circuit:  It will be the Default Gateway address for the Bugzilla group.
  4. service rules for Bugzilla servers are updates to reflect their new IP addresses.
Clients on the outside don't see a thing -- they are still happily talking to the CSS via virtual IP 198.41.30.X. However, on the inside, Bugzilla and "www" can now talk to each other using their load-balanced virtual IP 198.41.30.X since the CSS must be used to route all traffic between them.  If one node fails, the CSS continues to use the remaining nodes, and service remains functional for inside clients too.

Friday, February 15, 2013

Big Server Move Reason #4: Big Savings

This is the final part of a blog series on why moved to a new datacenter.

See Also: Reason #1: Bigger Pipe
See Also: Reason #2: Big Power
See Also: Reason #3: Big Cooling

Reason #4: Big Savings

The new colo facility was eager to have our business.  Very eager.  They kept sweetening the pot until it was practically impossible to say 'no'.  The end result of this move: more bandwidth, more AC power, better cooling, more cabinet space, and a lower monthly bill for the Foundation.

What's there not to like?