The Case of the Missing Translation

Since iWoman was acquired by FOKUS KIND Medien we have been working on a complete UI overhaul. This also requires many new translations. The previous version of iWoman had been localized with Linguan, a Mac app which we also had sold a while ago. Unfortunately Linguan does not yet have support for XLIFFs, even though the new owner is considering it.

This blog post describes an issue that the XLIFF workflow has because of a bug on POEditor.com as well as a problematic “feature” in Xcode which causes some strings to go missing.

At WWDC 2015 I was told by the guys responsible for localization at Apple that they consider strings files as a byproduct of the proper way to do localization this day. Apparently nowadays you don’t mess with strings files any more, but rather with XLIFF files. XLIFF is short for XML Localization Interchange File Format.

XLIFF solves an issue that has plagued the strings based workflow for quite some time: it also contains information of which file the localizations belong to. This way you can have multiple string tables contained in one XLIFF file per language and Xcode knows where a translation goes.

Extraction

The first step is to get Xcode to extract the localization tokens. You don’t even have to localize any files just yet. In this sample I have only set up a complex view controller and have not added any localization yet.

To get the XLIFF file, in Project Navigator, click on the project root and select the menu option Editor – Export for Localization …

Then select a place for the folder where Xcode should put the XLIFF files. You also have an option to select only the development language or to also include other languages (if you have added any). This got me an en.xliff file looking like on the following screenshot. Note the source text for the two paragraphs in the middle which I have highlighted.

You got to appreciate the beauty of the structure! Under the xliff root node you have a node for each file. And under each file you have a trans-unit for each string. Each translation unit has an id which matches the Object ID in Xcode. The source node contains the string you put into Interface Builder and note contains some supplementary information, repeating text, ObjectID and the object’s class name.

As you can see I have two newlines between the two paragraphs in the longer introductory text. Those are also present in the source node and show escaped as \n in the note. This will become important for the mystery which is about to unfold.

Import to POEditor.com

When I asked around for recommendations about how to get these XLIFF files translated, POEditor.com was much praised and so we set up an account with them. The praise themselves as one of the very few who can even handle XLIFF files.

We got one of these now, let’s make a new project and get it properly imported. I created a new project via the POEditor dashboard:

As suggested by “To start, add a language” – I click the button to Add Language to add a language. My development language is English, so I add that at this point.

This new language does not have any “available terms” just yet, we need to import these from our en.xliff we produced in the first step. In the dark vertical bar on the right side, choose the upward arrow button which shows “Import Terms” once you hover over it.

A few options need to be set for our kind of XLIFF.

We don’t just want to import the tokens, we also want the English texts. The texts in the UI so far might not be the ones we want to show up in the final app, since they are developer texts. As we seen above there is nothing in the target nodes of our XLIFF file. This is why we choose to “get translations from <source> instead of <target> tag”.

Clicking “Import File” we get two messages:

The second is of particular importance. The project needs a default reference language because this is what translators will see as source text. It also is necessary to have a reference language set because otherwise the XLIFF files you will export later would have empty source tags. And this is the the second piece to the unfolding mystery.

So we click the “Ok, let’s do it” button to make English the default reference language. We see the English translations screen and here we witness a case of data corruption.

Comparing this text with the one highlighted in the original XLIFF, it becomes apparent that POEditor removed the newlines from the text imported from the source tag. Oh no!

I wonder why POEditor does not like newlines. In my case they saved me from having to do do two separate labels, one per paragraph. I would imagine that other developers think the same way.

Translate and File

Let’s pretend for now that all is well, not having any knowledge of what will follow. We add a new Language German and translate almost all the items there:

The translation shows the English reference text above ours and we fill in the German translation below it, again making a paragraph break with two newlines. The export button in the sidebar (downward array pointing to hard disk) gives us an “iOS XLIFF” file.

Let’s look at the resulting translation unit.

As we can see – plain as day if your nose it put right to it by an annoyed developer – the source node no longer contains any newlines but reflects the text as it showed in the English translations.

In an earlier experiment I had not set a reference language. In that case the source tag is present but empty. We also see that POEditor escapes newlines in translations to \n, also something that Xcode does not seem to do.

Which is correct? Should newlines in strings be escaped as \n or not? The XML does not care.

Incomplete Import

Continuing our assumption that we don’t know anything about the missing newlines, we can now simply import the German translation into Xcode. Note that we haven’t added any localization languages to Xcode so far. Editor – Import Localizations …

Xcode will show us a before and after picture. There it looks like the two paragraph text is shown at the right place. On the left side you have “No Translation”, on the right you have the new German text.

BUT … if we look at the newly generated Main.strings (German) we get disappointed. If you enable English localization for the Main.storyboard we see that we should have 5 translated items, but here we only have 4. But on this newly created strings file the multi-paragraph text is missing.

I experimented a bit on how I would have to change the German XLIFF so that the translation gets accepted by Xcode. It turns out that the source text needs to be exactly as it is in Interface Builder, including non-escaped newlines. Even if the newlines where there but \n\n then the translation would still be missing.

Conclusion

This mystery is solved! There are several issues that work together so that in the end some strings might simply be missing from your localizations…

Bug 1: POEditor.com should not change the source strings in any way. Especially it should not simply delete newlines as those might have some significance.

Bug 2: This depends on whether or not you should slash-escape source strings in XLiFF files. If yes, then this is a bug in Xcode as this is clearly missing. rdar://23410569

Bug 3: Xcode should not consider the source string for assigning translations to objects. The identifier of the translation unit should be the only key considered because it is likely that the source text will either be changed (to make it more user friendly) or carry different escaping coming back from the translation service. rdar://23410520

After reporting this to POEditor I received word back from them:

We’re aware of this case and we went through endless discussions about this when we implemented the .iOS XLIFF format. The documentation on XLIFF implementation is scarce at best and as a result we had to make a lot of research for this feature.

The good news is that POEditor does not ignore non-escaped new lines, but only if xml:space=”preserve” is set on the ‘file’ tag or the ‘trans-unit’ tag.

Since Apple doesn’t say anything about this, we had to rely on the general XLIFF standard (declared in the headers of the .iOS XLIFF as well) when we wrote the specs.

I’ve added this note to the Radar mentioned for Bug 2.

Categories: Apple, Bug Reports