DNS resolution and the disadvantages of gethostbyname

1. Preface:

 

In network programming, domain name conversion to IP is often required. In this case, domain name resolution is required. Domain name resolution is a vertical request process, such.

 

2. performance bottleneck of gethostbyname

 

The gethostbyname function in Unix/Linux is often used to query the IP address of a domain name from DNS. Due to recursive DNS queries, the gethostbyname function often times out when querying a domain name. However, this function does not set the timeout as the setsockopt or select function does for functions such as connect and read. Therefore, it is often a bottleneck in the program. Someone suggested a solution to use alarm to set the timing signal. If the timeout occurs, use setjmp and longjmp to skip the gethostbyname function (I have not tried this method and do not know how it works ).
Under multiple threads, gethostbyname is a more serious problem, that is, if the gethostbyname of a thread is blocked, other threads will be blocked at gethostbyname. When writing crawlers, I also encountered this problem that left me confused for a long time. All crawler threads are blocked at gethostbyname, leading to slow crawling speed. Google has been online for a long time, and no answer is found. Today, I found an e-book "mining the web-discovering knowledge from hypertext data" in the googlegroup in the lab. The following text is used to explain crawlers:

Specified clients for DNS resolution are coded poorly. most Unix systems provide an implementation of gethostbyname (the DNS Client API-application program interface), which cannot concurrently handle multiple outstanding requests. therefore, the crawler cannot issue has resolution requests together and poll at a later time for completion of individual requests, which is critical for acceptable performance. furthermore, if the system-provided client is used, there is no way to distribute load among a number of DNS servers. for all these reasons, revoke crawlers choose to include their own custom client for DNS name resolution. the Mercator crawler from Compaq System Research Center was ced the time spent in DNS from as high as 87% to a modest 25% by implementing a custom client. the ADNS asynchronous DNS client library is ideal for use in crawlers.
In spite of these optimizations, a large-scale crawler will spend a substantial fraction of its network time not waiting for HTTP data transfer, but for address resolution. for every hostname that has not been resolved before (which happens frequently with crawlers), the local DNS may have to go into SS resume network hops to fill its cache for the first time. to overlap this unavoidable delay with useful work, prefetching can be used. when a page that has just been fetched is parsed, a stream of hrefs is extracted. right at this time, that is, even before any of the corresponding URLs are fetched, hostnames are extracted from the href targets, and DNS resolution requests are made to the caching server. the prefetching client is usually implemented using UDP instead of TCP, and it does not wait for resolution to be completed. the request serves only to fill the DNS cache so that resolution will be fast when the page is actually needed later on.

The general idea is that the gethostbyname of UNIX cannot be used in concurrent programs. This is a congenital defect that cannot be changed. Large crawlers often do not use gethostbyname, but implement their own customized DNS clients. In this way, DNS load balancing can be achieved, and the speed of DNS resolution can be greatly improved through asynchronous resolution. DNS clients often use UDP to resolve the IP address of a URL before crawling a webpage. The article also mentioned an Open Source asynchronous DNS library ADNS, the home page is http://www.chiark.greenend.org.uk /~ Ian/ADNS/
As can be seen from the above, gethostbyname is not applicable to multi-threaded environments and other programs that require high DNS resolution speed.

 

3. Method 1: Linux GNU gethostbyname_r

This method supports multithreading, And the standalone test can reach 100 times/s.

Parameter description: name -- indicates the Host Name of the webpage. For example, Baidu's host name is www.baidu.com.
RET -- if the result is successfully stored.
Buf -- this is a temporary buffer for storing various information in the process. Generally, the size of 8192 is enough. You can apply for an array char Buf [8192].
Buflen -- The Buf buffer size
Result -- if the operation succeeds, the hostent Pointer Points to ret, which is the correct result. If the operation fails, the result is null.
H_errnop -- store error codes
If the function is successful, 0 is returned. If the function fails, a non-0 number is returned.

Struct hostent {
Char * h_name; // official name of host
Char ** h_aliases; // Alias List
Int h_addrtype; // host address type -- af_inet | af_inet6
Int h_length; // length of address
Char ** h_addr_list; // list of addresses
};
# Define h_addr h_addr_list [0] // for backward compatibility

 

4. Method 2: write client requests by yourself

 

1. From the DNS message perspective, we mainly view the [QR] [opcode] [AA] [TC] [RD] [Ra] [(Zone)] In the DNS message header. [RCODE]
 
The main field of interest is the TC field. When the TC field is 1, it indicates that the total length of the response exceeds 512 bytes, and only the first 512 bytes are returned, in this case, DNS needs to use TCP to resend the original query request. Because in UDP applications, the application is limited to 512 bytes or smaller, so the DNS packet passing through the data stream can only be 512 bytes, TCP can divide user data streams into some packet segments, so TCP can use multiple packet segments to transmit data streams that exceed 512 bytes or data streams of any length.
 
Most books only write DNS using UDP port 53, which is incomplete and may lead to misunderstandings. They think that DNS only uses UDP instead of TCP.
 
2. From the application perspective, TCP and UDP are used for regional transmission. What is regional transmission? The DNS specification specifies two types of DNS servers, one is the primary DNS server and the other is the secondary DNS server. In a zone, the primary DNS server reads the DNS data from the local data file, while the secondary DNS server reads the DNS data from the authoritative DNS server. When a secondary DNS server is started, it needs to communicate with the primary DNS server and load data information, which is called zone transfer ). generally speaking, TCP is used for transmission between DNS servers, while UDP is used for transmission between the client and DNS server.