For your open source projects you want to make sure that pull requests you merge don’t destabilise the project. Having a large number of unit tests in combination with a Continuos Integration platform like Travis-CI helps greatly. With this setup each pull request or new branch sent to your repo triggers a build and Travis-CI will let you know if all tests pass or not.
If you annotate your headers with appledoc then you can build nice class documentation viewable that integrates with the Xcode documentation viewer or can be put on a web server as HTML.
Now what about pull requests which add new functionality but which are lacking some documentation annotations? This blog post looks into using Travis-CI for making sure documentation is also complete on pull requests.
On several occasions I found myself annoyed by my documentation build process – running on a private Jenkins CI – failing because a new parameter was missing the documentation. Docs get auto-built whenever there is a new commit in the develop branch and it will fail if there is something missing.
On all my open source projects – hosted on GitHub – I have appledoc set up such that if you build the Documentation target it builds the docs in HTML and as a docset. The latter is what you can integrate into Xcode.
The configuration for the docs building is contained in AppledocSettings.plist which contains information which folders to scour for headers to retrieve documentation from, which programming guide documents to include as well as some meta info, like project name and version.
The Documentation target consists only of a Run Script Build Phase calling the appledoc binary and specifying the output folder.
Naturally this requires the appledoc binary to be present in /usr/local/bin which incidentally is the place where the install-appledoc.sh – found at the root of the appledoc project – is placing it into.
This setup is hugely convenient for adding missing documentation because after a CMD+B build you see warnings in the code wherever they apply. So you can add these, rinse and repeat.
The guys from Travis-CI are so kind to preinstall appledoc on all Mac build nodes and this makes the whole thing possible.
In the context of a CI environment appledoc exists with status 0 only if there were no warnings nor errors. If there is even a single warning then it will exit with status 1. This causes Travis-CI to assume that the build command failed and consider the entire build to be a failure.
Simple Docs Test
I found two ways to have this occur. The simple method is to call appledoc and specify any temporary folder to write the output to. For example this is the .travis.yml file for DTCoreText.
--- language: objective-c before_script: - sudo easy_install cpp-coveralls script: - xctool project DTCoreText.xcodeproj -scheme DemoApp build test -sdk iphonesimulator6.1 -arch i386 ONLY_ACTIVE_ARCH=NO - xctool project DTCoreText.xcodeproj -scheme "Mac Framework" test -arch x86_64 ONLY_ACTIVE_ARCH=NO - appledoc -o /tmp . after_success: - ./coveralls.rb --extension m --exclude-folder Demo --exclude-folder Test --exclude-folder Externals
If the appledoc step fails (exit code not zero) then the after_success commands will not be executed.
Schema-based Docs Test
The second method is to actually use the same “Documentation” target that I mentioned previously because this is already set up correctly to call appledoc and uses the same configuration plist. Here’s the .travis.yml file modified to use the schema instead.
--- language: objective-c before_script: - sudo easy_install cpp-coveralls script: - xctool project DTCoreText.xcodeproj -scheme DemoApp build test -sdk iphonesimulator6.1 -arch i386 ONLY_ACTIVE_ARCH=NO - xctool project DTCoreText.xcodeproj -scheme "Mac Framework" test -arch x86_64 ONLY_ACTIVE_ARCH=NO - xctool -project DTCoreText.xcodeproj -scheme "Documentation" after_success: - ./coveralls.rb --extension m --exclude-folder Demo --exclude-folder Test --exclude-folder Externals
You need to make the schema shared for this to also work on other people’s machines including the Travis-CI build nodes. All it takes is a check mark and then to add the resulting xcshareddata/xcschemes/Documentation.xcscheme to the git repo. Without this step there will be no Documentation scheme available for xctool to be run.
In this screenshot you see that all the schemes I am referencing from the yml file have the checkmark in the Shared column. This is why.
I was worried for a second that the appledoc binary might be in a different location on the build nodes, but it turns out that this works just the same. It is there in /usr/local/bin too.
Testing Incomplete Documentation
Now to verify that our test actually works, let’s make the documentation intentionally miss the documentation for any parameter. I removed the documentation line for one parameter and when I hit build on the Documentation scheme the warning about the missing detail shows up at the right location.
Until now I didn’t really care about the return status of the Run Script build phase described above. But for the purpose of this documentation testing we need to have the build system consider the build as failed if there is any warning emitted. To pass through the exit status from the appledoc run we need to tweak the script such that it takes the return value of appledoc – contained in the environment variable $? – and exit with that.
echo "Documentation Output directory: ${BUILD_DIR}/Documentation/"
/usr/local/bin/appledoc --print-settings --output "${BUILD_DIR}/Documentation/" "${PROJECT_DIR}"
exit $?
The exit needs to occur right after appledoc because otherwise the echo command would reset the return value, and an echo rarely fails. With this tweak the Documentation build shows as failed if there is even a single warning, exactly what we need. If somebody would send me a pull request with thusly incomplete documentation I would see that in the pull request:
Clicking on the link would show the reason for the failure. It gets displayed by xctool a little less nicely than the other build steps but it serves the purpose.
With this information you know that you don’t want to merge this until the sender of the pull request has added the missing documentation.
Additional Checking via Xcode 5
There is an additional level of checking that can be added, as pointed out to me by Luis Solano on Twitter. Xcode 5 gained the ability to work with documentation comments as well, and the comment style used by appledoc is also supported. You can turn this on in the Xcode build settings. This is equivalent to passing the -Wdocumentation setting.
As soon as I enabled this I got a few additional warnings that appledoc missed. You should enable this for a binary build and you can even combine it with the “Treat Warnings as Errors” settings to construct an equivalent to the above mentioned procedure.
This warning applies to all sub-targets of the target you are adding it to. When I enabled it on a project Xcode began to spew out warnings about missing documentation comments in a sub-project.
Travis-CI and Multiple Build Commands
I was under the false assumption that Travis-CI would consider a build as failed if any of the build steps was producing an error. A build is considered a failure if the last build command exits with a different status value than 0. The recommended workaround for this is to move the build commands into a shell script.
build.sh
#!/bin/sh set -e xctool -project DTCoreText.xcodeproj -scheme DemoApp build test -sdk iphonesimulator6.1 -arch i386 ONLY_ACTIVE_ARCH=NO xctool -project DTCoreText.xcodeproj -scheme "Mac Framework" test -arch x86_64 ONLY_ACTIVE_ARCH=NO xctool -project DTCoreText.xcodeproj -scheme "Documentation"
The set -e causes the shell to exit if any subcommand or pipeline returns a non-zero status, which is exactly when one of these build steps fails. The build.sh needs to be executable and the yml file changes to:
--- language: objective-c before_script: - sudo easy_install cpp-coveralls script: - ./build.sh after_success: - ./coveralls.rb --extension m --exclude-folder Demo --exclude-folder Test --exclude-folder Externals
Conclusion
With the technique demonstrated you can make sure that documentation quality does not deteriorate from merging pull requests into your open source project. You can either call appledoc from the build script – provided you have the configuration contained in a plist – or call it from a Run Script phase in an Xcode target.
You can turn on documentation validation in Xcode 5 to gain an additional level of verification. This is useful because my tests have shown that Xcode 5 is somewhat stricter in validation than appledoc appears to be. I had some typos in the documentation tags which appledoc did not see anything wrong with, but Xcode properly reported as incorrect. Another advantage is that somebody making changes might not have appledoc installed. Turning on -Wdocumentation will validate documentation comments even in this situation.
The final thing I learned from this experiment relates to the way Travis-CI determines if a build should be considered as failure. If you have multiple build commands you have to group them in a shell script. Otherwise building would continue after a failed build part.
Categories: Recipes
