Curl usage Problems

Curl must be the preferred tool for web page capturing and simulated login development. Thanks to its powerful functions, it also provides support for multiple platforms. Recently, I used curl to simulate HTTPS website login. I encountered some problems. The truth is really hard to find, because I am not familiar with the network. Later, after debugging, find the cause.

Curl download: http://curl.haxx.se/download.html

The official website provides software and dynamic library download multiple platforms, here I use QT development, so the use of Windows platform mingw compiled version: http://curl.haxx.se/gknw.net/7.28.1/dist-w32/curl-7.28.1-devel-mingw32.zip

Decompress the package and you can see the include, Lib, bin and other directories. during compilation, use the DLL dynamic library under the bin directory, including the include header file, to compile and use the file. This version supports SSL and can access HTTPS websites.
For help documentation problems, you can download the curl source code, which provides the complete documentation, which is very detailed. If you want to use curl for development, the documentation in it is already very clear.

I encountered the following problems:
1. Failed to log on to the HTTPS website using curl (it is confirmed that the connection between URL encoding and socket is persistent .)
2. the source code of the obtained webpage is a little different from the source code accessed by the browser.

1.Log on to the HTTPS website. If you do not use the certificate, set the following:

curl_easy_setopt(m_pCurl, CURLOPT_SSL_VERIFYHOST, 0L);curl_easy_setopt(m_pCurl, CURLOPT_SSL_VERIFYPEER, 0L);

2.The web page code captured by curl is inconsistent with that opened by the browser, which is mainly caused by User-Agent settings. If you must use the browser format, you can directly use the User-Agent value of a browser to simulate web page access by the browser. Here I am using the User-Agent of Firefox browser:

curl_easy_setopt(m_pCurl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0");

3.URL Encoding
For the post operation, the submitted form fields must be converted to URL encoding. Some people are not familiar with URL encoding, but only convert some special characters into a common format.
For example, convert '=' to "% 3d"
If the function library is not provided, you can write a conversion function based on the URL encoding table.

4.Persistent connection

You can add or modify the desired option in the default HTTP header of curl. Here I want to add the persistent connection option: Connection: keep-alive:

struct curl_slist *list;list = curl_slist_append(list, "Connection: keep-alive");curl_easy_setopt(m_pCurl, CURLOPT_HTTPHEADER, list);

Set it to curlopt_httpheader to add it.

5.If you encounter a strange problem with curl, you 'd better enable the debugging option and combine the HTTP Fox plug-in of Firefox

Curl_easy_setopt (m_pcurl, curlopt_debugfunction, curldebug); // print the complete debugging information curl_easy_setopt (m_pcurl, curlopt_verbose, 1); // print the debugging information

6.Here we will share a post operation for a ready-made curl:

Bool ccurl: Post (const qstring & actionurl, const qstring & fieldsinfo, qstring & htmlstr) {curlcode code; struct curl_slist * List; List = curl_slist_append (list, "connection: keep-alive "); curl_easy_setopt (m_pcurl, curlopt_url, actionurl. tolatin1 (). data (); curl_easy_setopt (m_pcurl, curlopt_ssl_verifyhost, 0l); curl_easy_setopt (m_pcurl, scheme, 0l); curl_easy_setopt (m_pcurl, curlopt_coo Kiejar, m_cookiesfilename.toascii (). data (); curl_easy_setopt (m_pcurl, curlopt_cookiefile, m_cookiesfilename.toascii (). data (); curl_easy_setopt (m_pcurl, expires, writetomem); curl_easy_setopt (m_pcurl, curlopt_writedata, & htmlstr); curl_easy_setopt (m_pcurl, curlopt_postfields, fieldsinfo. tolatin1 (). data (); curl_easy_setopt (m_pcurl, curlopt_post, 1l); curl_easy_setopt (m_pcurl, curlop T_followlocation, 1l); curl_easy_setopt (m_pcurl, curlopt_httpheader, list); curl_easy_setopt (m_pcurl, curlopt_useragent, "Mozilla/5.0 (Windows NT 5.1; RV: 17.0) gecko/20100101 Firefox/17.0 "); // curl_easy_setopt (m_pcurl, curlopt_debugfunction, curldebug); // print the complete debugging information // curl_easy_setopt (m_pcurl, curlopt_verbose, 1 ); // print the debugging information code = curl_easy_perform (m_pcurl); If (curle_ OK! = Code) {qdebug () <"curl_easy_perform:" <curl_easy_strerror (CODE); curl_slist_free_all (list); Return false;} curl_slist_free_all (list); Return true ;}

7.Debugging functions can be written in this way, which is found on the Internet:

int CCurl::CurlDebug(CURL *pcurl, curl_infotype itype, char * pData, size_t size, void *){    if(itype == CURLINFO_TEXT)    {        qDebug() << "[TEXT]:" << pData;    }    else if(itype == CURLINFO_HEADER_IN)    {        qDebug() << "[HEADER_IN]:" << pData;    }    else if(itype == CURLINFO_HEADER_OUT)    {        qDebug() << "[HEADER_OUT]:" << pData;    }    else if(itype == CURLINFO_DATA_IN)    {        qDebug() << "[DATA_IN]:" << pData;    }    else if(itype == CURLINFO_DATA_OUT)    {        qDebug() << "[DATA_OUT]:" << pData;    }    return 0;}