Travis CI For Pull Request Validation

If you have an project that is accepting code contributions then you will want to make sure that those well-meaning additions don’t break existing functionality. Services like GitHub allow you to review such pull requests via their colored diffs or even give instructions as to how to manually pull the commits into a local branch for evaluating this.

But there is a limit that is quickly being reached if you need to check all those pull requests manually. Even worse, this is a mundane task, usually trying out if the targets build and running the unit tests. Repetitive tasks like this are boring and thus nobody can be faulted for starting to neglect them.

In this blog post I will explain how to set up Travis-CI to have all pull requests be automatically checked for you. You will know for each pull request if merging it into your develop branch would break the build or unit tested functionality.

I have to admit that from my own experience of recent past.

You Get Sloppy

DTCoreText has one that runs on iOS Simulator and one that runs on Mac. Furthermore on Mac I am executing all iOS unit tests, but built with 64-bit Mac architecture plus I am am comparing the output for a set of html files between Mac’s built-in initWithHTML: method and DTCoreText.

I went back over the 11 most recent version releases of DTCoreText. Checked out the release tag, updated submodules and ran both unit tests. As you can see I got sloppy with 1.6.x, half of these releases should not have been made with failing unit tests. Not a good sign of good software quality.

• 1.5.0: all pass
• 1.5.1: all pass
• 1.5.2: all pass
• 1.5.3: all pass
• 1.6.0 – iOS: all pass, Mac: 3 fail
• 1.6.1: all pass
• 1.6.2: all pass
• 1.6.3: iOS: app pass, Mac: build failure
• 1.6.4: all pass
• 1.6.5: iOS: all pass, Mac: 1 fail
• 1.6.6: all pass

You might argue that since you only care about iOS the Mac unit tests are not relevant to you. But it turned out that there were several scenarios where there simply as no unit test to catch a problem on iOS. It’s easy to have all green unit tests if you don’t have enough tests.

The Value of Unit Testing

Unit Tests are only as good as their “code coverage”. And they are only as good as they are run often enough to alert you about problems.

Code coverage is a measure of how many lines of your code are being touched by your unit tests. Say if you only unit test 10% of your classes, then your coverage will be below 10%. Naturally this means that if there is a bug in the other 90% then you won’t find it.

This is why I am pushing people to add unit tests for all bugs they find … unfortunately until recently I was the only person actually creating these. But if you are contributing to an Open Source project that has unit tests, then you should also contribute unit tests that test what you have implemented.

Proponents of “test-driven development” do it the other way around: they write the tests first, seeing them fail and then they implement the minimum necessary code to have the test pass. Then they write more tests and so on.

And even if you have great coverage then you still need to have some automated process run the unit tests for you. Or even better to also have it run on pull requests before they are able to “taint” your pristine code.

Enter Travis CI

There are several solutions to automating build and unit testing. All have some Continuous Integration (CI) system that is able to build and test projects. Travis CI is the most popular one for Open Source projects and it also integrates incredibly well with GitHub.

Generally speaking most CI solutions out there are written in some language like Java that allows them to run on any Unix hardware. Building iOS or Mac apps requires a Mac-based computer. While the management part of CI can run anywhere there needs to be an agent or worker running on Mac OS X to do the actual building.

Travis CI announced Mac, iOS and RubyMotion Testing support for Travis CI in April 2013. This is possible because they have OS X agents running on Sauce Labs’s Mac cloud platform. This allows Travis CI to run on standard unix hardware and integrate with GitHub while the actual work is done on Macs.

Travis’ Open Source plan is free to “use fairly”, perfect for using this for all your Open Source Projects. There are multiple – outdated – guides to be found on the Internet, let’s see what we need to do to set up a project.

Set Up Troubles

I already successfully set up Travis CI for DTCoreText, it only took me 13 tries. Some of these where caused be there being no simple to use and actually working setup guide. You can forget about the default build script because this fails on code signing, and you can forget about instructions that will have you install components with brew.

All of this is obsolete, the modern way to set up Travis CI is to use Facebook’s xctool which improves on xcode-build on many levels, not the least of which is way nicer build logs.

For this guide I shall set up another Open Source project of mine that has unit tests: DTFoundation. This way I can make sure that I am including all the necessary steps.

Step 1 – Prepare a Scheme

xctool (and xcode-build for that matter) works best with schemes. So your first order of business is to create and/or configure a schema. Schemes are basically instructions for the compiler to execute certain common tasks, like building or testing.

Any schema we want to be able to build needs to be shared and committed to the source repository. By default schemas are specific to the user, but if you share them they are available to other users of the project.

For our most basic setup we want to build the iOS target “Static Library” for the “Build” action and for “Test” we want to configure all unit tests configured in the “Unit Tests” target. We can add more schemes later, but to get started we want to see that something builds and that all our unit tests are being executed.

To get the shared schemes to not be ignored by git you might have to modify or temporarily disable your gitignore. You want all contents of xcshareddata/xcschemes/ underneath your xcodeproj bundle.

bigdrops:DTFoundation oliver$ git status
# On branch develop
# Your branch is ahead of 'origin/develop' by 15 commits.
#   (use "git push" to publish your local commits)
#
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#	new file:   DTFoundation.xcodeproj/xcshareddata/xcschemes/Static Library.xcscheme
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   .gitignore
#

We commit this xcscheme to the repository so that Travis can pick it up later.

Step 2 – Add a Build Script

We are going to build and test this scheme with xctool. Travis CI needs to have a file named .travis.yml present in the project root folder. For more complex setup requirements you can specify scripts to run before and after the build. We simply execute xctool specifying all it needs to know.

language: objective-c

script: xctool -project DTFoundation.xcodeproj -scheme "Static Library" build test -sdk iphonesimulator -arch i386

Step 3 – Connect Travis CI and GitHub

You go to travis-ci.org/profile and after signing in with your GitHub account you see a list of all your public repositories. There you simply flip the switch on the repository you just set up.

If you flip the switch to ON then Travis CI will add itself as a service hook for the GitHub repository. You can check this setup by clicking on the small wrench icon. Should something go wrong you can manually set up Travis by entering your GitHub user name and token which you can find on your Travis account profile. The domain field should be blank.

Step 4 – Fine Tune

Right after you made the connection you will see a build number 1 appear on your Travis CI home page, which also serves as your dashboard. You will probably have to fine tune the project settings as well as the xctool parameters a little until you get a build and test to go through.

In my case I kept getting a confusing message

No architectures to compile for (ONLY_ACTIVE_ARCH=YES, active arch=x86_64, VALID_ARCHS=i386).

This was caused by me having several static library targets set to Build Active Architecture Only: YES for Debug builds. Setting it to default NO fixed the problem and so I finally had a build go through.

At this point Travis CI will trigger a build whenever you push to any branch on the repository, or when there is a new pull request.

Step 5 – Certifying Pull Requests

Now it is awesome to have Travis test every commit of yours, but even more awesome is to have it inspect pull requests. If somebody sends you a pull request that causes a test to fail the pull request on GitHub will inform you about that. You also will see if there is an ongoing build related to this pull request, in which case you should wait a few more seconds for that to finish.

This is how that looks like, for testing I made a modification that would break some functionality covered by a unit test on DTCoreText:

I then proceeded to fix the pull request which caused the “Good to merge” to appear.

Honestly I cannot stop being fascinated by the fact that Travis is able to modify the look on the GitHub pull request like that.

Conclusion

Having a good set of unit tests will allow you to judge the risk of merging new pull requests at a glance. The more work you invest in good code coverage of your tests, the less areas remain where pull requests can break existing functionality.

Being able to connect Travis CI for free to your Open Source project and having it test builds and perform unit tests is invaluable. Key is the fact that you don’t have to do any manual work any more to run the tests. If anything then you have more time available to actually write those unit tests you were putting off until now.

Categories: Recipes