Thursday, August 14, 2014

Working Jenkins

Well, that was a lot of trial and error.

So, here is how the Jenkins setup is working. I wrote a script (available at at this repo in my github account) called maintain_firefox_cache.sh. (It calls other scripts in the checkout directory). What it does:

- Calls mozdownload. It will download the latest nightly build, but the name does not stay the same from day-to-day.
- If this is the first time it is downloaded, it will copy the download payload to "firefox-latest-lighty.en-US.<platform>.<ext>", where <platform> and <ext> are appropriate to the platform we are caching.
- If there is one there, it uses the unix find command to find the name of the latest binary, and copies that one. It also finds binaries older than the cached version and removes them.
- On the Mac, this runs in a Mac builder, and this script will open the .dmg, copy the contents out, and repackage them into a .tar.bz2 file, since the steeplechase machine will be on Linux and doesn't easily know how to open a .dmg file.

This leaves an artifact on the Jenkin's master filesystem. This is important later on.

We also have to have the payload (this is the Firefox 34 version) with the tests directory. We are using the linux64 version, because our tests do not required any platform-specific compiled assets from the package. Unfortunately, mozdownload does not know about the tests payload, and this url will have to updated everytime there is a version bump. Maybe I'll add that to mozdownload some day.

So that's great. We have firefox binaries, and test assets. Both of these need to be on the local filesystem of the steeplechase machine. So, how do we get them there?

My previous post talked about a couple of plugins designed to help track assets and when they changed:

  • URLSCM is really good at copying assets based on URLs. It's polling mechanism is broken, however; it always wants to download the asset even when it has not changed.
  • URLTrigger allows you to track modification date changes, but does not actually copy them.
For the tests download, I use the URL Trigger to detect changes and URLSCM to download it.

For the Firefox binaries, I was trying to use URL Trigger to track changes in the URL of the Last Successful builds, but they were never triggering. Instead, I use the FSTrigger to detect when files change on the Jenkins master itself.

So there is the sequence (using linux64 as an example):

  1. Once every 24 hours, firefox-nightly-linux24 fires. It run maintain_firefox_cache.sh, which runs mozdownload to get the linux64 binary. If there is a new binary, the new firefox-latest-nightly.en-US.tar.bz2 file is archived.
  2. trigger-firefox-nightly-linux64 (running on the Jenkins master) notices the new file, and immediately triggers expand-firefox-nightly-linux64.
  3. On the steeplechase machine, expand-firefox-nightly-linux64:
    1. Copies the firefox-latest-nightly-en-US.tar.bz2 file from the Jenkins master to the local filesystem.
    2. Expands the payload to the /home/mozilla/firefoxes/nightly/linux64 directory.
    3. Triggers all of the steeplechase jobs based on linux64 to be run.
  4. A steeplechase job will run, passing the correct binaries and test files to the test machines.
I also have add the SCM Sync plugin to the Jenkins instance so hopefully I won't have to create all of these jobs on the ESX machine from scratch (although I will have to edit them).


Monday, August 11, 2014

Back to Jenkins fun

So, now, I know how to download binaries and get the correct versions independent of what their names actually are.

Now, I need my jobs to trigger each other. The scheme I have is:


  1. Run download. If the binary is newer, overwrite the canonically named version, i.e., firefox-latest-nightly.en-US.linux-x86_64.tar.bz2.
  2. Another job triggers when firefox-latest-nightly.en-US.linux-x86_64.tar.bz2. It runs on the Steeplechase machine, and it expands the archive into a known directory location.
  3. It then triggers jobs for all of the steeplechase runs that depend on it.
At Coverity I used the URLSCM plugin to do the triggering. Basically it used the SCM polling mechanism builtin to Jenkins to see if the local copy of a file is newer than the version at a URL. The problem is, this mechanism broke a few years ago, and to this day nobody has fixed the bug.

Today I found out about another Jenkins plugin, URLTrigger Plugin. This allows you to trigger builds of a variety of things, but most germain to me is that it can trigger if md5 checksums are different. I am trying this out overnight; we'll see what happens.


Thursday, August 7, 2014

Which Firefox build to download?

So, eventually this system is going to have to download all Firefox releases and test them. All of the releases are available here:

http://ftp.mozilla.org/pub/mozilla.org/firefox/

So, there are a LOT of releases on this server. So which ones to we want? Mozilla has five (or more) active release trains at any one point:


  • Nightly - This is a nightly build of the code checked into mozilla-central, the Mercurial repository that holds Firefox. It is the least stable of them all.
  • Aurora - This is for code that is Feature Complete™, or some other terminology, which means that the feature has landed but has had very little testing.
  • Beta - We want to release this build next time we release something, so we are going through final testing.
  • Latest - This is the latest release. Right now, this is Firefox 31, but that will change next release cycle.
  • ESR (Extended Support Release) - This release is primarily intended for large organizations who want fewer releases for stability. Right now, there are two of these, FF 24, and FF 30. More info on this topic is here.
There is a utility called mozdownload which is here. Once you build and install it, mozdownload is a big help. It deals with changing file names and dates and the like.

So where is all of this stuff on ftp.mozilla.org? And what is the mozdownload command-line for it?

I hope that this helps. It sure helps me.