Xcode Coverage

Code Coverage is described on Wikipedia as:

… a measure used to describe the degree to which the source code of a program is tested by a particular test suite. A program with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a program with low code coverage.

Less bugs? Yes, please!

The first big step to take on your road to better software quality you have to start implementing unit tests. Those are short programs that test a specific class to make sure that individual methods do what they are supposed to, for the majority of possible states. But how do you get started?

One good approach I found is to create a unit test for each reported issue before proceeding to actually fix it. This gives you the confidence in the future that you are not accidentally un-fixing a previously killed bug. Even better is the approach called Test-driven Development. Hereby you are formulating assumptions about the end-results should be first in the form of unit tests. Then you proceed to implement the simplest possible solution for each test.

As the number of unit tests grow there is a point when you can no longer know from glancing at the tests if they are thorough enough. This is where you would start to measure Code Coverage. This employs readily-available tools to eliminate the guesswork.

Code Coverage

You can only measure code coverage for a program. i.e. something needs to run and a tool needs to keep track which lines of code are actually getting executed and how often. This sounds a bit like the CPU time Instrument where you can see which lines of code need the most CPU time. This is done by taking a sample at fixed intervals and accumulating the CPU time for each line of code.

Code Coverage measurement is simpler, it basically just has a counter for each line that gets increased whenever execution passes it. Since Xcode 4.3 you have all you need to produce coverage reports at your fingertips.

There are two ingredients:

1. Xcode needs to create a mapping file that marks lines and blocks in your source code
2. During execution of your unit tests xcode needs to produce an execution report

Both parts are achieved by two compiler flags that produce two corresponding file types as output. Actually these flags have been in existence much longer than clang (LLVM) is the main compiler used by Apple. Both of them had already been present in gcc, the original gnu C compiler, and when clang was created its parents made sure that it inherited all the useful flags and options from gcc.

The two flags to go with the above two ingredients are:

1. The .gcno file is generated when the source file is compiled with the GCC -ftest-coverage option. It contains information to reconstruct the basic block graphs and assign source line numbers to blocks.
2. The .gcda file is generated when a program containing object files built with the GCC -fprofile-arcs option is executed. A separate .gcda file is created for each object file compiled with this option. It contains arc transition counts, and some summary information.

(source: CoverStory Wiki)

Of course you don’t need to manually specify those flags any more since they are accessible via build settings. Enable the “Generate Test Coverage Files” in the “Apple LLVM 5.0 – Code Generation” section.

Enabling this section will cause the compiler to also generate gcno files. You need to specify this for all code involved in the coverage analysis, i.e. static library as well as unit tests. Enabling this option adds instrumentation code to the produced binary as well as slows down compilation enormously. For that reason you don’t want to have it enabled by default.

You enable the “Instrument Program Flow” setting, also found in the “Apple LLVM 5.0 – Code Generation” section.

By setting both options to Yes you have done all the configuring necessary. This is only to show you where these settings can be found, for the actual setup I’ll explain a better way below.

Coverage Output

To see if the settings are actually working, we can do a simple test. First be build just the Static Library target. The fastest way to get to the folder holding the gcno files is this:

1. Right-click on the static library product in the Products group, Show in Finder.
2. Right-click on the Debug-iphoneos in the header and pick the Build folder
3. Intermediates
4. PROJECT.build
5. Debug-iphonesimulator
6. TARGET.build
7. Objects-normal
8. i386

The full path is:

~/Library/Developer/Xcode/DerivedData/PROJECT-HASH/Build/Intermediates/PROJECT.build/Debug-iphonesimulator/TARGET.build/Objects-normal/i386/

Looking at this folder you will see 4 files for each class implementation, amongst them a gcno file.

To see the runtime instrumentation gcda  files being produced we run the Unit Test target on iPhone Simulator.

Note: At the time of this writing Xcode 5 is unable to run Unit Tests on 64-bit iOS Simulator (rdar://15107216). You need to run it any of the 32-bit simulators instead.

Note 2: Also at the time of this writing running an instrumented unit test on iOS 7 Simulator will not produce any gcda files (rdar://15107228). As a simple workaround you can run your unit tests with iOS 6.1 Simulator. There is also a workaround available, documented on Stack Overflow.

Note 3: The gcov tool bundled with Xcode seems to have some issues reading a few of my gcda files produced by the iOS Simulator (rdar://15107741). You also get lots of warnings mentioning “version ‘404*’, prefer ‘402*'” and that the source file could not be found. The same file, produced by a Mac-based unit test works fine. Go figure.

The reason for iOS 7 Simulator failing to produce these files is that the files are only written when the __gcov_flush() function is being called. Apparently due to some change in iOS 7 this call is missing. The workaround is to create a subclass of SenTestLog and call the flush function ourselves at the end of testing.

@interface DTCoreTextTestLog : SenTestLog
@end
 
// GCOV Flush function
extern void __gcov_flush(void);
 
static id mainSuite = nil;
 
@implementation DTCoreTextTestLog
 
+ (void)initialize
{
	[[NSUserDefaults standardUserDefaults] setValue:@"DTCoreTextTestLog" forKey:SenTestObserverClassKey];
 
	[super initialize];
}
 
+ (void)testSuiteDidStart:(NSNotification *)notification
{
	[super testSuiteDidStart:notification];
 
	SenTestSuiteRun *suite = notification.object;
 
	if (mainSuite == nil)
	{
		mainSuite = suite;
	}
}
 
+ (void)testSuiteDidStop:(NSNotification *)notification
{
	[super testSuiteDidStop:notification];
 
	SenTestSuiteRun* suite = notification.object;
 
	if (mainSuite == suite)
	{
		// workaround for missing flush with iOS 7 Simulator
		__gcov_flush();
	}
}
 
@end

When everything is working, you will see gcda files appear after the tests have been run. Contrary to the gcno files which belong to the tested code object files, the gcda files will appear next to the object files for the unit test, e.g. Debug-iphonesimulator/UnitTest.build/Objects-normal/i386/

Configuration Woes

In writing this up I hit a slight bump at this point. I found that you need to enable both compiler settings for all tested code. For a static library target that would mean that you would have the coverage-emission code compiled into the binary, even for release builds. If you have a static library with coverage enabled and try to link that into an app without those settings you get linker errors:

The reason is that adding the coverage options to the static library added calls to llvm_gcov functions to the lib. But not having these options for the app means that it wouldn’t link in the necessary dynamic library for the code which in turn causes the linker to be missing these symbols.

The best workaround I found was to create a new configuration, next to the already existing Debug and Release configurations. This enables us to enable the coverage options for everything without affecting debugging or releasing.

At the project root of your targets, add a Coverage configuration, based on Debug.

Now we remove the coverage flags for all targets and set them on the Coverage config instead.

To have this configuration be active for our Unit Test scheme, we have to choose it of course:

This causes the output files to be in a different place now. Instead of Debug-iphonesimulator everything now appears in Coverage-iphonesimulator.

Having a Look with CoverStory

GCDA is a binary format for which we need to have a viewer application to make sense of. For the Mac the most prominent viewer is CoverStory. You can download a DMG of the latest version and just put it into your Applications folder.

CoverStory needs the gcov-4.2 tool to process the coverage files. You may need to download and install the Commmand Line Tools for your Xcode and OS X version from Apple. In my case I had a working gcov binary in /usr/bin so I soft-linked that to gcov-4.2.

Since coverage relates to a specific unit test case, you will have to go through the individual GCDA reports one by one. Of course it does not make any sense opening the unit test’s GCDA files as you will have 100% coverage there, all code lines in the unit tests are usually executed.

If you inspect any tested code then you will see ranges in red where execution never occured:

The coverage reports cover all executable lines in the source files, but often there are methods which do not need to be tested. I’ve seen two approaches online to deal with those.

1. Add a COVERAGE preprocessor define and then enclose those non-relevant code sections in an #ifndef block
2. If you are going to use CoverStory, there are some special defines for that: To mark a single line as non-feasible, add a // COV_NF_LINE to the end of the line. To mark a block as non-feasible, surround it with // COV_NF_START and // COV_NF_END.

Unfortunately  CoverStory does not seem to be very stable and on several files it does not seem to be able to find some source files, especially headers. If you know how I can fix that, please let me know. I think that this might be related to the above mentioned issue in note 3, since CoverStory is running gcov as well behind the scenes.

Online Coverage Analysis with Coveralls

Having it all configured and producing the coverage output is only half the fun if you are doing that only for yourself. Especially for an open source project with tons of unit tests already it makes sense to use Coveralls. This is a web-based service – not unlike Travis-CI – which is willing to receive coverage reports via their API.

While Travis is getting notified from GitHub whenever you push, Coveralls is waiting to receive the coverage data in their specific JSON format via HTTP post.

Very few people seem to write unit tests and even fewer Objective-C developers seem to be doing coverage analysis. Which is probably why there is no ready-to-use solution to be found for us. After having run an instrumented unit test via Travis you need to collect the gcda and gcno files, process them with the gcov tool (bundled with Xcode) and then reformat them suitable for Coveralls.

I found only 2 people using Coveralls with Objective-C code (Masonry and Block Injection). Both are employing a Python script cpp-coveralls which does the necessary post-processing. The problem both are facing is how to find the folders containing the gcda files. One does that by running a post building script that exports the environmental variables into a shell script. The other does some trickery trying to convince xctool to give him the correct values.

My approach is different from theirs. You get a fresh build node every time Travis-CI builds your project and thus I don’t fuss with trying to determine the correct project’s folder. Instead I am finding all files in the Derived Data folder of the current user. And to make it a fun exercise this is the first Ruby script I wrote. You can find it here. Any additions to make it more sturdy are welcome!

Having enabled the project on Coveralls you see the coverage report appear as soon as you had a successful build and that triggered an upload of the coverage data.

Rather than showing you a couple of screenshots, you can have a look for it yourself. On this link you see a file that is covered fairly well at 91% but you see a couple of red blocks. These are the untested code you might not have known that you are not testing it.

One big advantage of collecting coverage reports on a web-service like Coveralls is that it can inform you about change in coverage level over time as well as for specific pull requests. A nice feature is that Coveralls can add a comment to pull requests.

You can see that GitHub, Travis-CI and Coveralls play together rather nicely for pull requests. The unit tests that Travis runs on the pull request inform me if I can merge without danger. Coveralls rounds of the picture letting me know if additions might be dramatically decreasing my code coverage. In that case I can ask the submitter to add a few tests for the new code he committed.

Conclusion

Having your compiler produce gcno and gcda files is the necessary input for any kind of Cover Coverage analysis. For open source projects we find that the combination of unit tests, run by Travis-CI on each push for every branch and coupled with code coverage analysis give us some great tools to improve the quality of our software.

Knowing what parts of your code don’t get executing during running an app or performing a unit test can give you insight as to what code might have become obsolete or inspire the creation of  additional unit tests.

Seasoned pros would tell you that attaining 100% coverage is unnecessary and useless. You shouldn’t write unit tests for overwritten -description methods for all your objects. Though – personally – I feel that seeing a number in the coverage badge has the distinct magnetism of trying to push that ever higher. Sort of a gamification aspect for unit tests. Having something inspire you to make your code slimmer, remove non-essential parts or test the untested can only be a good thing.

Finally, wouldn’t it be great if you could view the code coverage (like in Cover Story) inside Xcode? There’s a Radar for that, dupe it if you approve: rdar://15121260

Categories: Recipes