Crawler's mobile app capture tutorial-Pro-Test HTTP and HTTPS can be achieved

Now many websites have to do their own app-side products, a good crawler engineers, must be able to bypass the difficult crawl points and take shortcuts, which is happy. But in the online acquisition and access to countless documents and materials, I test countless times, can not normally get HTTPS data, the reason is that the phone-side certificate installation is not correct, so write articles to help later small partners.

Now grasping contractor has Fiddler,charles,wireshark, mobile phone app most commonly used is fiddler and Charles, and wireshark suitable for testing, the following I focus on the use of Charles grab Bag tool.

Charles is the artifact in the grab Bag tool that developers can use to easily crawl HTTP requests, HTTPS requests, and support modifications, replay, and more. It is very convenient for both PC and mobile terminals.

1. Install Charles:

can be downloaded anywhere on the internet, here do not introduce

2. After installing Charles, a simple configuration is required:

　　Note: To make sure that the phone and Charles are in the same WiFi environment

Use Ipconfig to see the IP that your computer uses to connect to WiFi:

　　

Open Charles, click Proxy->proxy Settings in the upper left corner and select Enabel transparent httpproxying. Click OK. Port is generally 8888, which is the same as the port in the next phone

　　

3, the configuration of Charles after the need for mobile phone-side configuration.

Turn on your phone, set->wifi--> to connect to your phone wifi--> Click Modify Network--show advanced selection--select agent as Manual---hostname fill in the IP address you just ipconfig, port number 8888, Click Save. OK, phone-side settings

　　

4, restart Fiddler, at this time can crawl to the HTTP packet, if you only need to crawl HTTP data, at this time has been implemented, if you need to get HTTPS packet, please continue to look down

　　

HTTPS request for access, need to make a certificate for mobile phone installation

1. For example, choose to install the Charles root certificate on your mobile device:

　　

Download Then

Then click on the second install under ssl_proxying, such as:

　　

2. Then open the phone's browser, enter the address in the back of browse to download, install it

Normally, the HTTPS packets can be crawled normally at this time, if the HTTPS packet is not crawled properly at this time, do not worry, there is a trick : There is no normal crawl here because your phone does not have a normal installation certificate, here we use analog Android simulator to replace the phone to grab the bag, I use the "Daily simulator", download the daily simulator on the Internet, and install, after installation, configuration steps with the phone. After the configuration is complete and the certificate is successfully installed, I can enjoy the HTTPS capture process of I. If you encounter other problems or can not catch the package, you are welcome to leave a message.

Fiddler Grab Bag Tool Tutorial reference documentation:

https://kb.cnblogs.com/page/130367/

http://www.jianshu.com/p/9e05a2522758

Http://www.cnblogs.com/zhaoyanjun/p/7068905.html

Crawler's mobile app capture tutorial-Pro-Test HTTP and HTTPS can be achieved