How to Spy on the Web Traffic of any App

Have you ever wondered what is going on when all those apps on your iPhone communicate with websites and web services? In this article I will explain a technique to employ your Mac as a spy to be able to inspect all the traffic that goes on between the public Internet and your iPhone.

This is wonderful for learning what POST requests need to be made of if you do screen scraping. It’s also quite useful if you are planning to reverse engineer some API that is not public yet. Finally you can use it to look for potential security concerns to report to the makers of your favorite apps.

You require a Mac that has a wired internet connection as well as built-in WiFi. We’ll use the Mac as the “Man in the Middle” and route all Internet traffic from our iPhone over it so that we can inspect the HTTP/HTTPS.

Create an Ad-Hoc Wireless LAN

First we create an ad-hoc network on our Mac. Right-Click on the WiFi symbol in the task bar and “Create Network”

If you want to you could choose an encryption but we won’t bother.

You will now see an ad-hoc symbol instead of the signal strength and your Mac will show as connected to the network. Next we need to share our wired Internet connection over the ad-hoc WLAN.

Share Your Wired Internet with the WLAN

So we go into the system settings and open the Sharing options.

We check Internet Sharing and see that we are sharing our connection from Ethernet to computers using AirPort, just like we want it to. The WiFi symbol changes again, now it looks like an upward pointing arrow. And actually however funny we choose our WLAN name does not matter because the Mac renames the Ad-Hoc network to be the same as the computer, in my case: iDrops.

Connect iOS Device to Ad-Hoc WLAN

Now we want to see if this setup works and we get actual Internet connectivity via our Mac. So we enter Settings, WiFi and connect to the WLAN we just created.

You should see via the status bar that you are staying connected. If this does not want to stay on the WiFi connection i.e. reverts to the cellular data icon, then disable cellular data via General – Network and switching cellular data off. This forces all data to go via WiFi. But usually it should be staying happily connected to your Ad-Hoc WiFi.

To prove that Internet Connectivity is there, just open mobile Safari to any website and see if it appears.

Network Debugging Proxy

There are many more or less geeky possibilities to inspect the traffic that’s now going via your Mac between the Internet and your iPhone. A free one is Wireshark. It is free, multi-platform and runs on Mac via X11. This works fine as long as you are only interested in HTTP traffic. But if you want to look under the hood of encrypted HTTPS then you need something stronger. I recommend the Charles Web Debugging Proxy, which is free to try (15 minutes at a time) and is definitely worth the purchase. Another disadvantage of Wireshark is that it is very difficult to set up on Mac because you need to fix some permissions as well as being an allround packet inspector and so it takes quite a bit of getting used to to be able to read HTTP with it. Charles is specialized in the kinds of traffic that you would see from a browser or applications using HTTP/HTTPS and thus saves you quite a bit of time.

So we installed Charles and launch the app. In menu Proxy – Proxy Settings you can see that Charles installed itself on port 8888.

It also automatically configured your networks to use itself as a proxy. This is the reason why you immediately see traffic appear in Charles.

Charles has two modes of viewing. The Structure mode groups requests by site, but I prefer the Sequence mode which shows connections sequentially. But we still don’t see any traffic from the device. The reason being, that while Charles is able to configure the machine it is running on, it has no influence over your iPhone.

So we need to do one further tweak there to also proxy via Charles the traffic from there. We enter our WiFi Settings and set a manual HTTP proxy. You enter the name of your Mac and the same port that Charles is using, in this case 8888.

If you have Charles running at that time you will get a warning there that actually tells you something good. Your iPhone is trying to use Charles to talk to the Internet. Just what we want, so we approve.

If you look in the WiFi settings of this connection you see that the stated address 10.0.2.4 is indeed the address of my device. This IP address will probably will be different for you, but still be the same between Charles and your iPhone.

You can always hit the trash bin icon in Charles’ toolbar to empty the list of connections. I often use that myself just before I trigger any kind of communication on the device because this way the following connections are right at the top of the list and I don’t have to scroll around looking for where it started. Constantly there will be new connections appearing just from the applications you have running on your mac, Dropbox, Twitter et al.

To get acquainted with how Charles displays the traffic lets first open a website like www.cocoanetics.com and see what we see.

We see the individual requests, the status (200 is ok), the method (GET) which host the request was made to, the path and a few other infos. If you select on connection you can view the request and response in different ways. Raw shows it like it really was sent. You can see that my User-Agent was the Safari on my Mac. If you open the same page in Mobile Safari, the only difference will be that the user-agent will be different, like Mozilla/5.0 (iPhone;…

So at this stage we are set for watching HTTP unencrypted traffic floating by. Encrypted traffic is still opaque to us, we don’t even see the method, just a CONNECT and the host usually being at port 443. What’s great about Charles here is that you can have it proxy HTTPS connections as well if you know how. In the Overview of such a secure connection you will have a Notes field: “SSL Proxying not enabled for this host: enable in Proxy Settings, SSL locations”.

So we go to Proxy – Proxy Settings – SSL and add the host we want to enable proxying for with full domain name and port, usually 443 for HTTPS.

Now if we go to the same site in Safari a new dialog pops up. There we can see that Charles actually created an HTTPS certificate on the fly to impersonate the target website. Charles created a *.cocoanetics.com SSL certificate which of course Safari does not automatically accept as valid because the Root Certification Authority for this site is Charles itself and thus not trusted.

We could simply trust this certificate for the future, but this option is not available on the iPhone, so we better get the Charles Root CA certificate so that we can install it everywhere we want this certificate to be trusted.

Installing the missing Root CA Certificate on Mac

The Charles CA Certificate Bundle is available on the Charles Website. It is a ZIP file containing a charles-proxy-ssl-proxying-certificate.crt.

If you double-click on the crt file in your download folder the keychain utility will open and ask you if you want to trust all certificates created from the Charles Root CA from now on.

And once we did that all communication with HTTPS sites that you have added to the list of servers to proxy for will appear in plain text. Instead of CONNECT and encrypted data you will see the actual GETs and POSTs.

On iPhone the installation of the Charles Root CA is slightly more difficult. If you open the secure site in Mobile Safari you are presented with a dialog to tell you about the unverified identity.

Again, we need to somehow tell our iPhone that we trust SSL certificates coming from Charles. So we need to install the Charles CA Root Certificate on the device.

Installing the missing Root CA Certificate on iPhone

You could now use the Apple iPhone Configuration Tool to bundle this certificate up. Or – much easier – you could simply mail the crt file to yourself. Luckily the iPhone knows what to do with a crt file.

And by tapping on the attachment we can simply install the new Root CA. You’ll get a warning “The authenticity of Charles Proxy SSL Proxying cannot be verified. Installing this profile will change settings on your iPhone.” But we are ok with that, so we install it to establish a new chain of trust.

And now if we open up the secure site from Mobile Safari we have the same effect as on our Mac. Everything appears as plain text as if there where no encrypting taking place. We’ve totally hacked it. 🙂

Sniffing, Spoofing, Spying

From here on forward we are set up to look under the hood of not just Mobile Safari, but actually any other app. Ever wonder what kind of connections the app store app makes? Let’s have a look!

So we add a *.apple.com host with port 443 to the list of hosts. To force a reset of the communication, we remove the app store from the task list. We empty the list of connections in Charles and then we tap the App Store icon. And we see that the innocent App Store app makes quite a few requests.

Actually only requests to p10-buy.itunes.apple.com where encrypted. Those are querying if you have any queued downloads for eBooks, apps or music. This actually tells us that the app store, itunes and iBooks app are internally using the same download mechanism/library. The requests to a1.phobos.apple.com are app icons and artwork. The ax.su.itunes.apple.com request checks which apps have updates available. And there are also two calls to metrics.apple.com where Apple is tracking what their users are looking at. The call to itunes.apple.com is the one that returns a plist describing the currently showing page on the app store.

What about the Mobile iTunes Connect App?

I actually found out all of what I am describing here because I wanted to see if we could piggy back on the API that Apple has established for use with the official iTunes Connect app. I was actually quite surprised to be able to see the clear text communication once I performed the above described setup.

However, Apple’s engineers are not dumb, as they added an additional security layer to their API, not relying solely on HTTPS to keep the communication safe. The mobile ITC app never communicates the username and password in the clear. Instead a very long and quite unbreakable hash is generated from the credentials and sent instead.

Because of this we might know the URL that the app communicates with, we might even see the whole traffic, but we can never hope to discover how the password hash is being generated. If anything then we can take this as a lesson in how to perfectly secure our own web services. HTTPS is a nice to have encryption layer, but you should never communicate confidential data in the clear.

Conclusion

In this article I have shown how easy it is to spy on live apps and web pages. You now have a tool at your disposal that is very powerful to do good or evil with. I implore you to use it for good. Personally I use Charles whenever I need to screen-scrape something. Like if you want to know the exact POST call Safari makes to a form so that you can simulate this in code running on an iPhone.

The second lesson from this article is to never rely on HTTPS encryption alone to keep your web service calls safe. If somebody really wants to see what you’re sending they can. Therefore you should refrain from sending login information in the clear, instead encrypt it into a very long string.

And if you’re from Apple please hear this: I congratulate you on making the Mobile ITC API safe from being reverse-engineered. How about making it a public web service that tool vendors like myself can code against? Then guys like me (with MyAppSales) wouldn’t have to use tools like Charles to scrape around in your new AJAXed iTunes Connect pages. That would save us much time.

Categories: Recipes