C + + implements image crawling on HTTPS Web pages

I. Main principles

By sending an HTTP request, we get the HTML source code of the target webpage, then we get the URL of the image through regular expression, and save all the pictures of the page to a folder, which is the whole process of the software.

Two. Concrete Practice

Now many sites are HTTPS protocol, but some or HTTP protocol, in fact, HTTPS is the security version of the HTTP protocol, equivalent to Http+ssl,ssl is between the HTTP application layer and the TCP Transport layer, and HTTP than HTTPS to send data requires SSL encryption, Then send. So we want to send the data to the server via HTTPS protocol, we need to go through these steps:

First we first and the server socket connection, and then the SSL and the socket socket created to bind, and then we send the data are sent via SSL, the following describes the specific process:

First, we'll start with a socket connection:

Establish TCP connection bool Connect () {//Initialize socket Wsadata wsadata;if (0! = WSAStartup (Makeword (2, 2), &wsadata)) return false;// Create Socket G_sock = socket (af_inet, sock_stream, 0), if (G_sock = = invalid_socket) return false;//convert domain name to IP address hostent *p = Gethostb Yname (G_host); if (p = = NULL) return false;sockaddr_in sa;memcpy (&sa.sin_addr, P->H_ADDR, 4); sa.sin_family = AF_ Inet;sa.sin_port = htons (443); if (Socket_error = = Connect (G_sock, (sockaddr*) &sa, sizeof (SOCKADDR))) return false; return true;}

　　

Https=http + SSL, so using OpenSSL to send requests to the HTTPS site and the second chapter of the socket to send HTTP is very similar, just to put the SSL layer on the native socket, the basic process is as follows:

A. WSAStartup the Winsock service.

B. Creating a socket socket

C. Connect connection service side

D. Establishing an SSL context

E. Building SSL

F. Binding SSL to a socket socket that was previously established

G. Ssl_write () sending data

H. ssl_read () Receive data

BOOL Ssl_connect () {//Register the error strings for Libcrypto & Libsslerr_load_bio_strings ();//initialization of SSL libraries, all algorithms for loading SSL , load all SSL error messages Ssl_library_init (); Openssl_add_all_algorithms (); Ssl_load_error_strings ();//New context saying we are a client, and using SSL 2 or 3sslContext = Ssl_ctx_new (sslv23_client _method ()); if (Sslcontext = = NULL) {err_print_errors_fp (stderr); return false;} Create an SSL struct for the Connectionsslhandle = Ssl_new (Sslcontext), if (Sslhandle = = NULL) {ERR_PRINT_ERRORS_FP (Stder R); return false;} Connect the SSL struct to our connectionif (! SSL_SET_FD (Sslhandle, G_sock)) {ERR_PRINT_ERRORS_FP (stderr); return false;} Initiate SSL Handshakeif (ssl_connect (sslhandle)! = 1) {ERR_PRINT_ERRORS_FP (stderr); return false;} return true;}

Three. Problems encountered

1. First the VS console encoding is GBK, but some Web pages are UTF-8 so we're going to do that.

String UTFTOGBK (const char* UTF8) {int len = MultiByteToWideChar (Cp_utf8, 0, UTF8,-1, NULL, 0); wchar_t* wstr = new wchar_t [Len + 1];memset (wstr, 0, Len + 1); MultiByteToWideChar (Cp_utf8, 0, UTF8,-1, WSTR, len); len = WideCharToMultiByte (CP_ACP, 0, Wstr,-1, NULL, 0, NULL, NULL); c har* str = new Char[len + 1];memset (str, 0, Len + 1); WideCharToMultiByte (CP_ACP, 0, Wstr,-1, str, len, NULL, NULL); if (WSTR) delete[] Wstr;return str;}

Installation of 2.OPENSSL

Online has a lot of themselves to download the source code and then compile their own, but in this project we can directly use someone else compiled libraries do not compile themselves, directly in the VS project directory to add

3. There is the image of the download, is this type of LPCWSTR data, we use a string to convert to LPCWSTR (do not know why the urldownloadtofile inside of the two are to do this conversion can not use other functions)

String savepath = "e:\\c++_file\\ network crawler 1\\ network crawler \ \ Network crawler \\img\\" +to_string (i) + ". jpg"; size_t len1 = Savepath.length (); WCHAR _t* Imgsavepath = new Wchar_t[len1];int Nmlen1 = MultiByteToWideChar (CP_ACP, 0, Savepath.c_str (), Len1 + 1, Imgsavepath, L EN1);

Here is the complete code for the download:

　　　　　　　　   URL generated per = Mat[1].str (); size_t len = Per.length ();//get string length int nmlen = MultiByteToWideChar (CP_ACP, 0, Per.c_str (), Len + 1, NULL, 0);//If the function succeeds, and Cchwidechar is 0//The return value is the size of the wide character number required to receive the buffer for the string to be converted. wchar_t* buffer = new Wchar_t[nmlen]; MultiByteToWideChar (CP_ACP, 0, Per.c_str (), Len + 1, buffer, nmlen);//save path string Savepath = "e:\\c++_file\\ web crawler 1\\ crawler \ \ Web crawler \\img\\ "+to_string (i) +". jpg "; size_t len1 = Savepath.length (); wchar_t* imgsavepath = new Wchar_t[len1];int Nmlen1 = MultiByteToWideChar (CP_ACP, 0, Savepath.c_str (), Len1 + 1, Imgsavepath, len1); cout << mat.str () << Endl;cout & lt;< savepath << endl;//download file HRESULT hr = Urldownloadtofile (NULL, buffer, Imgsavepath, 0, NULL); if (hr = = S_OK) {Co UT << "-------OK" << Endl;}

Four. Complete code

#include "spider.h" int main () {cout << "*********************************************************" << Endl;cout << "*********************** Crawl picture system ***********************" << endl;cout << "************* << endl;//Create a picture of the stored directory CreateDirectory (L "./img", NULL);// Start fetching string StartURL = "https://www.shiyanlou.com/#sign-modal"; Startcatch (StartURL);//while (1); return 0;} void Startcatch (String starturl) {queue<string> Q;q.push (StartURL), while (!q.empty ()) {string cururl = Q.front (); Q.pop ();//Parse Urlif (false = = Analyse (Cururl)) {cout << "Parse URL failed, error code:" << GetLastError () << Endl; Continue;} Connect server if (false = = Connect ()) {cout << "Connection server failed with error code:" << GetLastError () << endl;continue;} Establish SSL connection if (false = = Ssl_connect ()) {cout << "Establish SSL connection failed with error code:" << GetLastError () << endl;continue;} Get Web page string html;if (false = = gethtml (html)) {cout << "get Web page data failed with error code:" << getlasterRor () << endl;continue;} if (false = = Regexiamage (html)) {cout << "get Web page data failed with error code:" << GetLastError () << endl;continue;} cout << html << Endl;} Release Ssl_shutdown (Sslhandle); Ssl_free (Sslhandle); Ssl_ctx_free (Sslcontext); closesocket (G_sock); WSACleanup ();}  Parse Urlbool analyse (string url) {Char *purl = new Char[url.length () + 1];strcpy (PURL, Url.c_str ()); char *pos = Strstr (PURL, "https://");//Find the string if (pos = = NULL)//At the beginning of http://false;else pos + = 8;//will omit the sscanf (POS, "%[^/]%s", G_host, G_o bject);d elete[] Purl;return true;} Establish TCP connection bool Connect () {//Initialize socket Wsadata wsadata;if (0! = WSAStartup (Makeword (2, 2), &wsadata)) return false;// Create Socket G_sock = socket (af_inet, sock_stream, 0), if (G_sock = = invalid_socket) return false;//convert domain name to IP address hostent *p = Gethostb Yname (G_host); if (p = = NULL) return false;sockaddr_in sa;memcpy (&sa.sin_addr, P-&GT;H_ADDR, 4); sa.sin_family = AF_ Inet;sa.sin_port = htons (443); if (Socket_error = = Connect (G_sock, (sockaddr*) &sa, sizeof (SOCKADDR))) return False;return true; BOOL Ssl_connect () {//Register the error strings for Libcrypto & Libsslerr_load_bio_strings ();//initialization of SSL libraries, all algorithms for loading SSL , load all SSL error messages Ssl_library_init (); Openssl_add_all_algorithms (); Ssl_load_error_strings ();//New context saying we are a client, and using SSL 2 or 3sslContext = Ssl_ctx_new (sslv23_client _method ()); if (Sslcontext = = NULL) {err_print_errors_fp (stderr); return false;} Create an SSL struct for the Connectionsslhandle = Ssl_new (Sslcontext), if (Sslhandle = = NULL) {ERR_PRINT_ERRORS_FP (Stder R); return false;} Connect the SSL struct to our connectionif (! SSL_SET_FD (Sslhandle, G_sock)) {ERR_PRINT_ERRORS_FP (stderr); return false;} Initiate SSL Handshakeif (ssl_connect (sslhandle)! = 1) {ERR_PRINT_ERRORS_FP (stderr); return false;} return true;}   BOOL Gethtml (string & HTML) {char temp1[100];sprintf (TEMP1, "%d", 166); string c_get; C_get = c_get+ "Get" + G_object + "http/1.1\r\n" + "Host:" + g_host + "\ r \ n" + "content-type:text/html; Charset=utf-8\r\n "//+" content-length: "+ Temp1 +" \ r \ n "//+" user-agent:mozilla/5.0 (Windows NT 10.0; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/58.0.3029.110 safari/537.36 edge/16.16299\r\n "+" connection:close\r \n\r\n ";//+ temp; Ssl_write (Sslhandle, C_get.c_str (), C_get.length ()), char buff[101];int nreal = 0;while ((nreal = Ssl_read (SslHandle, Buff) > 0) {buff[nreal] = ' + '; HTML + = UTFTOGBK (buff);//printf ("%s\n", Buff); memset (buff, 0, sizeof);} printf ("%s\n", HTML); return true;} String UTFTOGBK (const char* UTF8) {int len = MultiByteToWideChar (Cp_utf8, 0, UTF8,-1, NULL, 0); wchar_t* wstr = new wchar_t [Len + 1];memset (wstr, 0, Len + 1); MultiByteToWideChar (Cp_utf8, 0, UTF8,-1, WSTR, len); len = WideCharToMultiByte (CP_ACP, 0, Wstr,-1, NULL, 0, NULL, NULL); c har* str = new Char[len + 1];memset (str, 0, Len + 1); WideCharToMultiByte (CP_ACP, 0, Wstr,-1, str, len, NULL, NULL); if (WSTR) delete[] Wstr;return str;} BOOL Regexiamage (string & HTML) {Smatch Mat;regex rgx ("sRc=\ "(. * (png|svg|jpg)) \" ") string::const_iterator start = Html.begin (); String::const_iterator end = Html.end (); string Per;int i = 1;while (Regex_search (Start, End, Mat, RGX)) {//url generates per = Mat[1].str (); size_t len = Per.length ();//Get character string length int nmlen = MultiByteToWideChar (CP_ACP, 0, Per.c_str (), Len + 1, NULL, 0);//If the function succeeds, and the Cchwidechar is 0//The return value is the ease of receiving the string to be converted The size of the wide number of characters required to flush the area. wchar_t* buffer = new Wchar_t[nmlen]; MultiByteToWideChar (CP_ACP, 0, Per.c_str (), Len + 1, buffer, nmlen);//save path string Savepath = "e:\\c++_file\\ web crawler 1\\ crawler \ \ Web crawler \\img\\ "+to_string (i) +". jpg "; size_t len1 = Savepath.length (); wchar_t* imgsavepath = new Wchar_t[len1];int Nmlen1 = MultiByteToWideChar (CP_ACP, 0, Savepath.c_str (), Len1 + 1, Imgsavepath, len1); cout << mat.str () << Endl;cout & lt;< savepath << endl;//download file HRESULT hr = Urldownloadtofile (NULL, buffer, Imgsavepath, 0, NULL); if (hr = = S_OK) {Co UT << "-------OK" << Endl;} start = mat[0].second;i++;} return true;} LPCWSTR Stringtolpcwstr (String orig) {size_t origsize = orig.length () + 1;const size_t newsize = 100;size_t Convertedchars = 0;wchar_t *wcstring = (wchar_t * ) malloc (sizeof (wchar_t) * (Orig.length ()-1)), mbstowcs_s (&convertedchars, wcstring, Origsize, Orig.c_str (), _ TRUNCATE); return wcstring;}

Here is the header file:

#pragma once#include <iostream> #include <Windows.h> #include <string> #include <queue># Include <regex> #include <urlmon.h> #include <openssl/rand.h> #include <openssl/ssl.h># Include <openssl/err.h> #pragma comment (lib, "Urlmon.lib") #pragma comment (lib, "Libeay32.lib") #pragma comment ( LIB, "Ssleay32.lib") #pragma comment (lib, "ws2_32")  //linked to Ws2_32.lib using namespace Std;char G_host[max_path];char G_object[max_path]; SOCKET G_sock; SSL *sslhandle; Ssl_ctx *sslcontext; BIO * bio;//starts fetching void Startcatch (string starturl);//parsing urlbool analyse (string URL);//connecting server bool Connect ();//Establishing SSL connection bool Ssl_connect ();//Get Htmlbool gethtml (string& html);//utf turn gbkstd::string utftogbk (const char* UTF8);//Regular expression bool Regexiamage (string & HTML); LPCWSTR Stringtolpcwstr (std::string orig);

　　

C + + implements image crawling on HTTPS Web pages