Blocking of gethostbyname in multi-threaded Environments

From: http://read.newbooks.com.cn/info/196629.html

 

The gethostbyname function in Unix/Linux is often used to query the IP address of a domain name from DNS. Due to recursive DNS queries, the gethostbyname function often times out when querying a domain name. However, this function does not set the timeout as the setsockopt or select function does for functions such as connect and read. Therefore, it is often a bottleneck in the program. Someone suggested a solution to use alarm to set the timing signal. If the timeout occurs, use setjmp and longjmp to skip the gethostbyname function (I have not tried this method and do not know how it works ).

Under multiple threads, gethostbyname is a more serious problem, that is, if the gethostbyname of a thread is blocked, other threads will be blocked at gethostbyname. When writing crawlers, I also encountered this problem that left me confused for a long time. All crawler threads are blocked at gethostbyname, leading to slow crawling speed. Google has been online for a long time, and no answer is found. Today, I found an e-book "mining the web-discovering knowledge from hypertext data" in the googlegroup in the lab. The following text is used to explain crawlers:

 

Specified clients for DNS resolution are coded poorly. most Unix systems provide an implementation of gethostbyname (the DNS Client API-application program interface), which cannot concurrently handle multiple outstanding requests. therefore, the crawler cannot issue has resolution requests together and poll at a later time for completion of individual requests, which is critical for acceptable performance. furthermore, if the system-provided client is used, there is no way to distribute load among a number of DNS servers. for all these reasons, revoke crawlers choose to include their own custom client for DNS name resolution. the Mercator crawler from Compaq System Research Center was ced the time spent in DNS from as high as 87% to a modest 25% by implementing a custom client. the ADNS asynchronous DNS client library is ideal for use in crawlers.

In spite of these optimizations, a large-scale crawler will spend a substantial fraction of its network time not waiting for HTTP data transfer, but for address resolution. for every hostname that has not been resolved before (which happens frequently with crawlers), the local DNS may have to go into SS resume network hops to fill its cache for the first time. to overlap this unavoidable delay with useful work, divfetching can be used. when a page that has just been fetched is parsed, a stream of hrefs is extracted. right at this time, that is, even before any of the corresponding URLs are fetched, hostnames are extracted from the href targets, and DNS resolution requests are made to the caching server. the divfetching client is usually implemented using UDP instead of TCP, and it does not wait for resolution to be completed. the request serves only to fill the DNS cache so that resolution will be fast when the page is actually needed later on.

 

The general idea is that the gethostbyname of UNIX cannot be used in concurrent programs. This is a congenital defect that cannot be changed. Large crawlers often do not use gethostbyname, but implement their own customized DNS clients. In this way, DNS load balancing can be achieved, and the speed of DNS resolution can be greatly improved through asynchronous resolution. DNS clients often use UDP to resolve the IP address of a URL before crawling a webpage. The article also mentioned an Open Source asynchronous DNS library ADNS, the home page is http://www.chiark.greenend.org.uk /~ Ian/ADNS/

As can be seen from the above, gethostbyname is not applicable to multi-threaded environments and other programs that require high DNS resolution speed.