Tuesday, April 7, 2015

How to build Platform QA Jenkins

So, I never have done a step-by-step to build the Jenkins instance that I have built. Knowing how to do it yourself might be important if you want to test jobs running in Jenkins.

First of all, decide on the host OS for Jenkins. I chose Ubuntu 14 for several reasons:

  • It's free.
  • It's easy to install on a virtual machine.
  • There are now incredibly easy-to-install debian packages that do all of the work of installing Java, installing Jenkins, setting up a Jenkins user, and starting Jenkins automatically on Linux. Updating Jenkins then becomes a matter of running updates like you would for other software.

Setting up the master VM

  • Install Ubuntu on a VM host, like VirtualBox or VMWare.
    • Give the VM at least 2 cores, 80 GB of disk space, and 4 GB of memory.
  • Follow the instructions here:
    • https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu
  • Install the following Plugins. First follow the following steps:
    • Navigate to your jenkins instance.
    • Click on "Manage Jenkins"
    • Click on "Manage Plugins". (This is the url http://localhost:8080/pluginManager on your VM)
    • Click on "Available". Install the following:
      • File System SCM
      • Filesystem Trigger Plug-in
      • GIT client plugin
      • GIT Parameter Plug-in
      • GIT plutin
      • GitHub API Plugin
      • Mercurial plugin
      • Multiple SCMs plugin
      • SSH Agent Plugin
      • SSH Credentials Plugin
      • SSH plugin
      • SSH Slaves plugin
      • URLTrigger Plug-in
      • Windows Slaves Plugin
      • Workspace Cleanup Plugin
    • Restart Jenkins.
Next Installment: Add a builder

Thursday, August 14, 2014

Working Jenkins

Well, that was a lot of trial and error.

So, here is how the Jenkins setup is working. I wrote a script (available at at this repo in my github account) called maintain_firefox_cache.sh. (It calls other scripts in the checkout directory). What it does:

- Calls mozdownload. It will download the latest nightly build, but the name does not stay the same from day-to-day.
- If this is the first time it is downloaded, it will copy the download payload to "firefox-latest-lighty.en-US.<platform>.<ext>", where <platform> and <ext> are appropriate to the platform we are caching.
- If there is one there, it uses the unix find command to find the name of the latest binary, and copies that one. It also finds binaries older than the cached version and removes them.
- On the Mac, this runs in a Mac builder, and this script will open the .dmg, copy the contents out, and repackage them into a .tar.bz2 file, since the steeplechase machine will be on Linux and doesn't easily know how to open a .dmg file.

This leaves an artifact on the Jenkin's master filesystem. This is important later on.

We also have to have the payload (this is the Firefox 34 version) with the tests directory. We are using the linux64 version, because our tests do not required any platform-specific compiled assets from the package. Unfortunately, mozdownload does not know about the tests payload, and this url will have to updated everytime there is a version bump. Maybe I'll add that to mozdownload some day.

So that's great. We have firefox binaries, and test assets. Both of these need to be on the local filesystem of the steeplechase machine. So, how do we get them there?

My previous post talked about a couple of plugins designed to help track assets and when they changed:

  • URLSCM is really good at copying assets based on URLs. It's polling mechanism is broken, however; it always wants to download the asset even when it has not changed.
  • URLTrigger allows you to track modification date changes, but does not actually copy them.
For the tests download, I use the URL Trigger to detect changes and URLSCM to download it.

For the Firefox binaries, I was trying to use URL Trigger to track changes in the URL of the Last Successful builds, but they were never triggering. Instead, I use the FSTrigger to detect when files change on the Jenkins master itself.

So there is the sequence (using linux64 as an example):

  1. Once every 24 hours, firefox-nightly-linux24 fires. It run maintain_firefox_cache.sh, which runs mozdownload to get the linux64 binary. If there is a new binary, the new firefox-latest-nightly.en-US.tar.bz2 file is archived.
  2. trigger-firefox-nightly-linux64 (running on the Jenkins master) notices the new file, and immediately triggers expand-firefox-nightly-linux64.
  3. On the steeplechase machine, expand-firefox-nightly-linux64:
    1. Copies the firefox-latest-nightly-en-US.tar.bz2 file from the Jenkins master to the local filesystem.
    2. Expands the payload to the /home/mozilla/firefoxes/nightly/linux64 directory.
    3. Triggers all of the steeplechase jobs based on linux64 to be run.
  4. A steeplechase job will run, passing the correct binaries and test files to the test machines.
I also have add the SCM Sync plugin to the Jenkins instance so hopefully I won't have to create all of these jobs on the ESX machine from scratch (although I will have to edit them).

Monday, August 11, 2014

Back to Jenkins fun

So, now, I know how to download binaries and get the correct versions independent of what their names actually are.

Now, I need my jobs to trigger each other. The scheme I have is:

  1. Run download. If the binary is newer, overwrite the canonically named version, i.e., firefox-latest-nightly.en-US.linux-x86_64.tar.bz2.
  2. Another job triggers when firefox-latest-nightly.en-US.linux-x86_64.tar.bz2. It runs on the Steeplechase machine, and it expands the archive into a known directory location.
  3. It then triggers jobs for all of the steeplechase runs that depend on it.
At Coverity I used the URLSCM plugin to do the triggering. Basically it used the SCM polling mechanism builtin to Jenkins to see if the local copy of a file is newer than the version at a URL. The problem is, this mechanism broke a few years ago, and to this day nobody has fixed the bug.

Today I found out about another Jenkins plugin, URLTrigger Plugin. This allows you to trigger builds of a variety of things, but most germain to me is that it can trigger if md5 checksums are different. I am trying this out overnight; we'll see what happens.

Thursday, August 7, 2014

Which Firefox build to download?

So, eventually this system is going to have to download all Firefox releases and test them. All of the releases are available here:


So, there are a LOT of releases on this server. So which ones to we want? Mozilla has five (or more) active release trains at any one point:

  • Nightly - This is a nightly build of the code checked into mozilla-central, the Mercurial repository that holds Firefox. It is the least stable of them all.
  • Aurora - This is for code that is Feature Complete™, or some other terminology, which means that the feature has landed but has had very little testing.
  • Beta - We want to release this build next time we release something, so we are going through final testing.
  • Latest - This is the latest release. Right now, this is Firefox 31, but that will change next release cycle.
  • ESR (Extended Support Release) - This release is primarily intended for large organizations who want fewer releases for stability. Right now, there are two of these, FF 24, and FF 30. More info on this topic is here.
There is a utility called mozdownload which is here. Once you build and install it, mozdownload is a big help. It deals with changing file names and dates and the like.

So where is all of this stuff on ftp.mozilla.org? And what is the mozdownload command-line for it?

I hope that this helps. It sure helps me.

Thursday, July 31, 2014


I have a Jenkins instance running the regression suite on Firefox nightly builds in all platform combinations of linux64, linux32, and macosx. The front page looks like this:

So, I have proof-of-concept. I am demoing this at our QA Work Week QA Fair later today. Should be fun!

Lots of work to do, however.

  1. I need to investigate using parameterized builds. Having to create and maintain a separate Jenkins job for each combination is painful (I have a lot of experience with this from Coverity, alas).
  2. Henrick Skupin has a Jenkins instance, and he has solved the problem of the nightly version numbers changing every month. Need to implement that.
  3. Need to add jobs for nightly versions connecting to Aurora, Beta, Release and ERS versions of Firefox.
  4. The test is not incredibly valid. I need to take the test additions done by Geo Mealer in our Sunny Day Environment and run the connections for 1 minute each. I really want to do the parameterization first so I don't have to update dozens of jobs.
  5. I need to put this in our ESX farm. However, I only want to do this when most of the above is done. I am going to set up Jenkins on the ESX farm, running linux64 regressions for now.
  6. I have got to get Jenkins and Steeplechase working for Windows. While I am in Mountain View this week, I do not have access to my Windows VMs. Will have to wait until I get home next week.
  7. Sunny Day Environment does most of its job maintenance and triggering via cron jobs. It would be nice to move this into Jenkins.
  8. Need to report results of test runs into Treeherder. This is a pretty big effort.
  9. And then, there is B2G and Android.
  10. Steeplechase changes
    1. Run with existing binaries but new profile. (or whatever; independent control)
    2. Send binary archives down instead of directories, and tell clients how to unpack them.
    3. Steeplechase should talk to treeherder.
Ought to keep me busy.

Wednesday, July 30, 2014

Another fun fact: For Negatus to be able to run firefox for the tests, it has to run it in a display. Without setting up a fake X session, the Negatus client has to be logged in, at least on Linux. I am sure that this will be true on other platforms as well. For Linux, we could run in a virtual frame buffer, but I am not sure that this is necessary. Just set up the account the test will run in to auto login. Make sure it is on a private network, though...

OK, it's just too hard to nail down all of the libraries to run firefox32 on linux64. I give up. Average users won't do this.