How to Monitor Mobile App Traffic With Sniffers

转自 https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/

by Oleg Nikiforov (@ddr3ams), QA Engineer at Stanfy • 13 min read

Are your QA engineers into monitoring apps’ traffic? Really, are they? The answer I usually hear is “why would I be?” Of course, this is irrelevant if your application works without access to a server and is purely offline. On the other hand, if your mobile app has network interactions, it is essential to make sure everything works smoothly there. Unfortunately, this part of testing is often used in web development but rarely used during mobile app development.
When I talk about analyzing app traffic I usually mean working with sniffers (that come in a form of proxy). These are special software that is intended to help you see network interactions in the form of HTTP or HTTPS requests and responses. While monitoring insecure HTTP traffic is relatively easy, working with HTTPS requires some additional actions (I’ll tell you about this in detail later).


Table of contents:
[Approaches to monitoring mobile application traffic](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Approaches to analyzing mobile app traffic)
[Setting up your workflow](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Setting up your workflow)
[How-to install a certificate on an Android](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#How-to install a certificate on an Android)
[How-to install a certificate on an iOS](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#How-to install a certificate on an iOS)
[Working with proxy](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Working with proxy)
[Optimizing your workflow with traffic sniffer](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Optimizing your workflow with traffic sniffer)
[When one proxy is not enough](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#When one proxy is not enough)
Summary

Approaches to analyzing mobile app traffic
Ask your mobile app developers what monitoring tool they use when developing the network interactions. The most common answer is “IDE console output”. Yes, it provides all the necessary information for you, and can even be used by your QAs (assuming they know how to build apps locally with the help of IDE :)), though it’s output can look really chaotic when server responses are large:

Android Studio output

Xcode output

And even when requests and responses are displayed on the console, they still may miss some important info, such as custom headers.
We’ve also made a PDF-version of this article. Download
While this approach can be used with applications you develop, you usually won’t be able to monitor other apps’ traffic. Indeed, there are cases when you have to analyze other applications (for example: when building your own API, analyzing other applications can help you to save time). This is where sniffers come in handy.
Let’s define the first advantage of using sniffers:
You get traffic analysis in a convenient and readable format

Burp Suite

In addition to simply reading network data, there are plenty of situations when you need to manipulate it. I assume you can find a way to inject needed data right into an application while debugging it if you know how to code. However, using sniffers for data manipulation is a relatively easier approach and you can change data without code knowledge. This brings us to the second advantage of using sniffers:
Easy data manipulation.

Setting up your workflow
There are several well known applications for sniffing HTTP/HTTPS traffic: Fiddler,Charles Proxy, TCP Catcher, Burp Suite. In my everyday work I use Burp Suite and TCP Catcher. It’s up to you what sniffer software to use, but further examples will be provided using Burp Suite, so to begin, I’ll tell you how to configure it. Setting-up a mobile sniffer for HTTP traffic is pretty easy:
Download Burp Suite – it’s a Java application, so you will need to install Java on your computer, in case you haven’t installed it already;
Run the app;
Open “Proxy” > “Options”: you will see a list of Proxy Listeners, one is already set by default;
Select it and click “Edit” > and you will see the next screen:

Edit proxy listener

4.1 Select a port to use with proxy (I use port 8200 but you can leave port 8080)4.2 Change the address to your IP4.3 Click “OK” and you’ll see something like this:

Proxy listeners

  1. Setup a proxy on your device.
    Let’s take a deeper look at how to setup a proxy on iOS and Android devices.
    How to set up a proxy on Android:

  2. Setting up a proxy for your Wi-Fi network: Go to “Settings” > “Wi-Fi” > long press on connected network > choose “Modify network” > change “Proxy” to “Manual” and enter your computer’s IP address into the “Proxy hostname” field and the port where your mobile traffic sniffer is running into “Proxy port”:

    Setting up proxy on Android device

  3. Click “Save“.
    How to set up a proxy on iOS:

  4. To set up a proxy on a device: Go to “Settings” > ”Wi-Fi” > open network > Scroll to “HTTP PROXY” section , change it to “Manual” and enter your computer’s IP into “Server” and your port where Burp is running into “Port”:

    Setting up proxy on iOS device

In Burp Suite, open “Proxy” > “Intercept” and turn interception off:

Turning off interception in Burp Suite

Next open the browser on your device and go to http://www.w3.org (or other site that uses HTTP protocol).
Finally, open “HTTP history” in the program and you should see requests there:
HTTP History" width="740" height="188" srcset="https://stanfy.com/wp-content/uploads/2016/04/9.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/9-300x76.png 300w, https://stanfy.com/wp-content/uploads/2016/04/9-1024x260.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">Proxy > HTTP History

Note: If you don’t see anything in this tab – check the “Alerts” tab for any Burp errors (the label becomes red if anything wrong happens). Also try to ping your device and check that your firewall is disabled or allows proxy connection; it is common to be unable to connect via proxy because of firewall settings.
We’ll talk about actual work with this data later, but first I’ll tell you how to set up HTTPS proxy. In order to be able to see and manipulate HTTPS requests you need to perform a MitM attack. While it’s not ethical to apply this to third-party applications, this is necessary for testing your own mobile app.

  1. Change Burp Suite certificate settings:
    Enable support invisible proxying
    Enable support invisible proxying
    Enable support invisible proxying

“Options” > “Proxy Listeners” > “Edit your listener” > “Certificate”" width="740" height="338" srcset="https://stanfy.com/wp-content/uploads/2016/04/11.png 1400w, https://stanfy.com/wp-content/uploads/2016/04/11-300x137.png 300w, https://stanfy.com/wp-content/uploads/2016/04/11-1024x468.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Proxy Listeners” > “Edit your listener” > “Certificate”

  1. Install CA on the device
    How to install a certificate on an Android device:
    Open the browser on your device and go to http://burp/cert and the certificate will be downloaded.
    Rename the downloaded file (cacert.crt) to cacert.cer (or any name with .cer).
    Go to “Settings” > “Security” > “Install from storage.
    Open cacert.cer to install it.

How to install a certificate on an iOS device:
Open the browser on your device and go to http://burp/cert > and the certificate will be downloaded and added to “Profiles.”

  1. On your device, open the browser and go to www.google.com (or other site that uses HTTPS protocol).
  2. Open “HTTP history” in Burp and you should see requests there:
    “HTTP History”" width="740" height="162" srcset="https://stanfy.com/wp-content/uploads/2016/04/12.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/12-300x66.png 300w, https://stanfy.com/wp-content/uploads/2016/04/12-1024x224.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “HTTP History”

After this, you still won’t be able to see the traffic of some applications. Mostly this is related to the usage of SSL pinning which prevents an app from trusting your certificate. Though this security check can be bypassed, that action is beyond the scope of this article.
Thus, if your application being tested is well protected, you should make changes in code or ask your developers for help. Below is the list of cases that could prevent you from sniffing traffic:
SSL pinning – can be turned off for debugging (make sure it is turned on for release build).
Apple Transport Layer Security – be aware that on iOS 9 you need to add additional parameters to use proxy: open Info.plist and add App Transport Security Settings > Allow Arbitrary Loads:YES
Okhttp library on Android may bypass device proxy settings – in that case you need to setup the proxy in source code (use OkHttpClient “setProxy” method):

client.setProxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("10.1.1.90", 8200)));

Working with proxy
I use version 1.6.32. Burp is updating regularly, but I don’t think this main flow should change in further updates.
This is how you can intercept requests and responses:
In Burp Suite open “Proxy” > “Intercept.”
Turn interception on.
Open Browser on device and go to www.google.com >. You’ll see an intercepted request:

On this step you can “drop” or “forward” this request

You can click on “Drop” to cancel this request or “Forward” to send it to the server.
OR you can intercept the response:

  1. Click on “Action” or click on request body with RMB > select “Do intercept” > select “Response to this request”:
    select “Do intercept” > select “Response to this request”" width="740" height="294" srcset="https://stanfy.com/wp-content/uploads/2016/04/14.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/14-300x119.png 300w, https://stanfy.com/wp-content/uploads/2016/04/14-1024x407.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">RMB > select “Do intercept” > select “Response to this request”

  2. Click on “Forward.”3. You will see the response and can modify it as you like. After all modifications are done, click on “Forward” to send request to the device:

    Click on “Forward” to send request to the device

Note: you should make any changes before you reach the timeout limit or this modified response won’t get to the device. Also check the “Options” > “Misc” > “Hotkeys” section in the app, they can save you plenty of time.
That is a basic workflow for using proxy.
Now I’ll show you some examples of how working with requests may help you in testing applications.

  1. Request validation
    Check that values submitted to the server are in the proper format.
  2. Response validation
    Check that the response is in the proper format and contains all necessary headers and data in the body.
  3. Long strings in fields
    Instead of submitting data to the server to test how long values are displayed you can simply change the value in the server response on the fly.
  4. Load more items
    When you have a “load more”, usually it is limited by a certain amount of items and this amount is passed on to the server; intercept the request and change this param to get more items and see how your application deals with a large amount of data.
  5. Testing error handling
    Intercept any server response and change its value to simulate error – 401 Unauthorized, 400 Bad Request, 500 Internal Server Error, and so on. Just make sure that responses are created properly; a good approach is to save real error messages for future usage.
  6. Test an expired session
    A good way to test an expired session is to intercept a request and change the value of token (or auth header), then the server will respond with a real error message.
  7. Test time-outs
    On iOS devices you can use Network Link Conditioner to simulate time-outs; on Android this task is a bit harder, but you can always use a proxy to test this – just intercept a request and wait until the app drops it because of the timeout.
  8. Perform actions limited by UI
    If you want to perform an action on some user but cannot do it because of UI limitation (this user isn’t shown to you) – perform this action on any other user, intercept the corresponding request and change the user ID to any value you need.
    Optimizing your workflow with traffic sniffer
    Burp Suite has various options to enhance your work with traffic:
    Some apps use various 3rd party libraries and may send tons of server requests that are not relevant for your tests. In this case you can add all needed URLs to scope and work only with them:

“Scope” > “Include in scope” - this is how you add certain URLs into scope" width="740" height="188" srcset="https://stanfy.com/wp-content/uploads/2016/04/16.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/16-300x76.png 300w, https://stanfy.com/wp-content/uploads/2016/04/16-1024x261.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Target” > “Scope” > “Include in scope” – this is how you add certain URLs into scope

“Http history” > tap on white bar to see available filters > toggle “Show only in-scope items” - this is how you apply scope filtering" width="740" height="224" srcset="https://stanfy.com/wp-content/uploads/2016/04/17.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/17-300x91.png 300w, https://stanfy.com/wp-content/uploads/2016/04/17-1024x310.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Http history” > tap on white bar to see available filters > toggle “Show only in-scope items” – this is how you apply scope filtering

Let’s compare how our history looks including all items and only in-scope items;

“All items” (pay attention to Filter)

Only items in scope

Don’t want to see certain requests occasionally? You can add them to the list of URLs excluded from scope and mark them when necessary:

“Scope” > “Exclude from scope” - this is how you remove certain URLs from scope." width="740" height="192" srcset="https://stanfy.com/wp-content/uploads/2016/04/20.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/20-300x78.png 300w, https://stanfy.com/wp-content/uploads/2016/04/20-1024x266.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Target” > “Scope” > “Exclude from scope” – this is how you remove certain URLs from scope.

Now look at our history that is filtered by scope while removing some items from it:

History that is filtered by scope while removing some items from it

You can configure app to intercept requests / responses when they meet certain conditions:

“Options” > “Intercept Client Requests” / “Intercept Server Responses” - while interception is turned on only items that apply to specified rules will be intercepted (be aware that you still have to intercept a request before intercepting a response)" width="740" height="423" srcset="https://stanfy.com/wp-content/uploads/2016/04/22.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/22-300x171.png 300w, https://stanfy.com/wp-content/uploads/2016/04/22-1024x585.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Intercept Client Requests” / “Intercept Server Responses” – while interception is turned on only items that apply to specified rules will be intercepted (be aware that you still have to intercept a request before intercepting a response)

You can also auto change any part of a response or request:

“Options” > “Match and Replace” - set rules for auto replacing content." width="740" height="164" srcset="https://stanfy.com/wp-content/uploads/2016/04/23.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/23-300x66.png 300w, https://stanfy.com/wp-content/uploads/2016/04/23-1024x227.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Match and Replace” – set rules for auto replacing content.

For example, if your application forces users to upgrade to a newer version that is available, this is usually triggered on the backend. You can change app-version (or any similar) header in all requests to see if the server responds with an error message.
To summarize usage of Burp:

When one proxy is not enough :)
Burp Suite is a great network monitoring tool for testing and debugging – I use it everyday and cannot imagine my current workflow without it. What I lack is the ability to log application traffic for 24 hours a day and on devices I don’t have access to (e.g. clients’ or beta testers’ devices).
To handle this necessity I use another proxy tool called Runscope – this is a cloud-based proxy. It doesn’t allow you to manipulate data but it can be useful for logging.
To set it up you need to change base URL in your mobile app to a string created by Runscope. In this way only your app will work with the proxy and only requests sent to this special URL will go through it. So if your app uses several URLs you will need to change all of them.

Use “URL Helper” to modify your base URL

By using Runscope you will slightly increase the time taken to perform client-server operations (as you are adding an additional point on the route of your data) but you get the ability to monitor app’s traffic from all devices.
Thus, when a client tells you that something isn’t working you can quickly check logs for any errors. Even when incorrect server responses may not trigger UI changes and be visible for users (for example registering for push notifications fails), by creating a filter for failed requests you will be able to check what is wrong at any point in time:
Search for “status: 400” to see requests that failed with 400 HTTP code:

Search for “status: 400” to see requests that failed with 400 HTTP code

Click on “Create Stream View” to save this filter in the “STREAM VIEWS” section:

Click on “Create Stream View” to save this filter in the “STREAM VIEWS” section

Now you will be able to quickly navigate to the list of requests that failed with 400 HTTP code:

Now you will be able to quickly navigate to the list of requests that failed with 400 HTTP code

Runscope pros and cons:

  • no need to turn on proxy on device+ proxy works for specified URLs– may be hard to setup (depends on which network library you use)– changes in source code are necessary to turn proxy on / off– free package is limited in number of requests (though it’s usually enough for testing purposes)
    Product Design and Development Tips & Infographics

Give it a try. It only takes a click to unsubscribe.

Summary
Analyzing app traffic doesn’t require any high-level skills though it may seem difficult when you begin. As you introduce this practice into your daily workflow, soon you will see how natural it is to use proxy every time you test your application.
I hope that the above examples will help you in mastering your mobile app testing skills and advancing them to the next level. Enjoy ;)
P.S. Although this article is written from the perspective of a QA Engineer, I strongly recommend that all mobile app developers who work on projects with a backend use sniffers for inspecting and manipulating apps’ traffic.
More interesting reads:
How Switching From Dev to QA Automation Changed The Way I Work
Tools and Platforms for Mobile App Analytics. Part I: Performance and Errors

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 151,829评论 1 331
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 64,603评论 1 273
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 101,846评论 0 226
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 42,600评论 0 191
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 50,780评论 3 272
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 39,695评论 1 192
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 31,136评论 2 293
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 29,862评论 0 182
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 33,453评论 0 229
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 29,942评论 2 233
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 31,347评论 1 242
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 27,790评论 2 236
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 32,293评论 3 221
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 25,839评论 0 8
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 26,448评论 0 181
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 34,564评论 2 249
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 34,623评论 2 249

推荐阅读更多精彩内容