Finally.

I have a Jenkins instance running the regression suite on Firefox nightly builds in all platform combinations of linux64, linux32, and macosx. The front page looks like this:

So, I have proof-of-concept. I am demoing this at our QA Work Week QA Fair later today. Should be fun!

Lots of work to do, however.

1. I need to investigate using parameterized builds. Having to create and maintain a separate Jenkins job for each combination is painful (I have a lot of experience with this from Coverity, alas).
2. Henrick Skupin has a Jenkins instance, and he has solved the problem of the nightly version numbers changing every month. Need to implement that.
3. Need to add jobs for nightly versions connecting to Aurora, Beta, Release and ERS versions of Firefox.
4. The test is not incredibly valid. I need to take the test additions done by Geo Mealer in our Sunny Day Environment and run the connections for 1 minute each. I really want to do the parameterization first so I don't have to update dozens of jobs.
5. I need to put this in our ESX farm. However, I only want to do this when most of the above is done. I am going to set up Jenkins on the ESX farm, running linux64 regressions for now.
6. I have got to get Jenkins and Steeplechase working for Windows. While I am in Mountain View this week, I do not have access to my Windows VMs. Will have to wait until I get home next week.
7. Sunny Day Environment does most of its job maintenance and triggering via cron jobs. It would be nice to move this into Jenkins.
8. Need to report results of test runs into Treeherder. This is a pretty big effort.
9. And then, there is B2G and Android.
10. Steeplechase changes
1. Run with existing binaries but new profile. (or whatever; independent control)
2. Send binary archives down instead of directories, and tell clients how to unpack them.
3. Steeplechase should talk to treeherder.

Ought to keep me busy.