Working Jenkins

Well, that was a lot of trial and error.

So, here is how the Jenkins setup is working. I wrote a script (available at at this repo in my github account) called maintain_firefox_cache.sh. (It calls other scripts in the checkout directory). What it does:

- Calls mozdownload. It will download the latest nightly build, but the name does not stay the same from day-to-day.
- If this is the first time it is downloaded, it will copy the download payload to "firefox-latest-lighty.en-US.<platform>.<ext>", where <platform> and <ext> are appropriate to the platform we are caching.
- If there is one there, it uses the unix find command to find the name of the latest binary, and copies that one. It also finds binaries older than the cached version and removes them.
- On the Mac, this runs in a Mac builder, and this script will open the .dmg, copy the contents out, and repackage them into a .tar.bz2 file, since the steeplechase machine will be on Linux and doesn't easily know how to open a .dmg file.

This leaves an artifact on the Jenkin's master filesystem. This is important later on.

We also have to have the payload (this is the Firefox 34 version) with the tests directory. We are using the linux64 version, because our tests do not required any platform-specific compiled assets from the package. Unfortunately, mozdownload does not know about the tests payload, and this url will have to updated everytime there is a version bump. Maybe I'll add that to mozdownload some day.

So that's great. We have firefox binaries, and test assets. Both of these need to be on the local filesystem of the steeplechase machine. So, how do we get them there?

My previous post talked about a couple of plugins designed to help track assets and when they changed:

• URLSCM is really good at copying assets based on URLs. It's polling mechanism is broken, however; it always wants to download the asset even when it has not changed.
• URLTrigger allows you to track modification date changes, but does not actually copy them.

For the tests download, I use the URL Trigger to detect changes and URLSCM to download it.

For the Firefox binaries, I was trying to use URL Trigger to track changes in the URL of the Last Successful builds, but they were never triggering. Instead, I use the FSTrigger to detect when files change on the Jenkins master itself.

So there is the sequence (using linux64 as an example):

1. Once every 24 hours, firefox-nightly-linux24 fires. It run maintain_firefox_cache.sh, which runs mozdownload to get the linux64 binary. If there is a new binary, the new firefox-latest-nightly.en-US.tar.bz2 file is archived.
2. trigger-firefox-nightly-linux64 (running on the Jenkins master) notices the new file, and immediately triggers expand-firefox-nightly-linux64.
3. On the steeplechase machine, expand-firefox-nightly-linux64:
1. Copies the firefox-latest-nightly-en-US.tar.bz2 file from the Jenkins master to the local filesystem.
2. Expands the payload to the /home/mozilla/firefoxes/nightly/linux64 directory.
3. Triggers all of the steeplechase jobs based on linux64 to be run.
4. A steeplechase job will run, passing the correct binaries and test files to the test machines.

I also have add the SCM Sync plugin to the Jenkins instance so hopefully I won't have to create all of these jobs on the ESX machine from scratch (although I will have to edit them).