[Gevent source code analysis] C-Ares asynchronous DNS requests

C-Ares is an asynchronous DNS request library. libcurl, libevent, and Wireshark all use C-Ares. Before gevent1.0, libevent is used,

Therefore, its DNS request uses C-Ares, and later versions use cython to encapsulate C-Ares.

C-Ares official documentation, http://c-ares.haxx.se/docs.html.

In gevent, DNS uses the thread pool version by default. You can use the C-Ares asynchronous library by setting the gevent_resolver = Ares environment variable.

How can we prove that it is indeed asynchronous? Just try to run it again?

#coding=utf8import socketimport geventfrom gevent import get_hubfrom gevent.resolver_ares import Resolverr = get_hub().resolver = Resolver(servers=[‘8.8.8.8‘])def f(w):    print w,r.gethostbyname(w)for w in [‘www.google.com‘,‘www.baidu.com‘,‘www.apple.com‘]:    gevent.spawn(f,w)gevent.sleep(6)

Let's talk about a few C-Ares library functions before getting started, and then look at ares. pyx with a friendly feeling.
Cares. ares_library_init (cares. ares_lib_init_all)
The Ares library is initialized only on the Windows platform. It is mainly used to load the lplpapi. dll and is not called on non-Windows platforms.
If the call is required, it must be called before any function of C-Ares.

Cares. ares_library_cleanup ()
Compared with cares. ares_library_init, The iphlpapi. dll will be released on the Windows platform, which is not called on non-Windows platforms.
This function is not called in gevent. The author also uses it in _ dealloc? I don't quite understand this. Maybe there is a better reason.

Cares. ares_destroy (self. Channel)
Destroy the channel, release the memory, and close the opened socket. This is called in _ dealloc _.

Cares. ares_init_options (& channel, & options, optmask)
This is the core function in Ares. It is used to initialize channel and options. optmask is mainly constructed through the _ init _ of channel.

cdef public class channel [object PyGeventAresChannelObject, type PyGeventAresChannel_Type]:    def __init__(self, object loop, flags=None, timeout=None, tries=None, ndots=None,                 udp_port=None, tcp_port=None, servers=None):

Parameter description:

Flags are used to control a query behavior. For example, ares_flag_usevc will send only TCP requests (we know that DNS has both TCP and UDP)
Ares_flag_primary: sends requests only to the first server. For other options, see the ares_init_options function documentation.
Timeout: specifies the time-out time for the first request, in seconds. The unit of C-Ares is millisecond. gevent is converted. The time-out period after the first time has its own algorithm in C-area.
Tries: Number of request attempts. The default value is 4.
Ndots: the minimum number of '.'. The default value is 1. If the value is greater than 1, the domain name is directly searched. Otherwise, the local domain name will be merged (init_by_environment sets the local domain name)
Udp_port, tcp_port: The UDP and TCP port used. The default value is 53.
Servers: the server that sends the DNS request. For details, see ares_set_servers.
Ndots said, for example, Ping Bifeng (a host of my colleague), and the detection finds that no '. '(that is, less than ndots), so the local region is added to the ares_search function in ares_search.c.

Cares. ares_set_servers (self. Channel, cares. ares_addr_node * c_servers)
After setting the DNS request server, you need to free up the memory space of c_servers, because the memory space in ares_set_servers is re-malloc.
In set_servers, the finally free memory space is used

            c_servers = <cares.ares_addr_node*>malloc(sizeof(cares.ares_addr_node) * length)            if not c_servers:                raise MemoryError            try:                index = 0                for server in servers:                    ...                c_servers[length - 1].next = NULL                index = cares.ares_set_servers(self.channel, c_servers)                if index:                    raise ValueError(strerror(index))            finally:                free(c_servers)

You may be curious about how C-Ares is associated with the gevent (libev) socket, because the nature of DNS is also
Therefore, the underlying layer also needs to use epoll and other mechanisms provided by the operating system, while C-Ares provides interfaces for socket status change,
This allows c-Ares to run on libev. All magic is actually provided externally by ares_options.sock_state_cb.

#ares.hstruct ares_options {  int flags;  int timeout; /* in seconds or milliseconds, depending on options */  int tries;  ....  ares_sock_state_cb sock_state_cb;  void *sock_state_cb_data;};ARES_OPT_SOCK_STATE_CB void (*sock_state_cb)(void *data, int s, int read, int write)

When the DNS socket status changes, sock_state_cb is called back, And gevent_sock_state_callback is set in _ init _ of the channel.

def __init__(...)    options.sock_state_cb = <void*>gevent_sock_state_callback    options.sock_state_cb_data = <void*>selfcdef void gevent_sock_state_callback(void *data, int s, int read, int write):    if not data:        return    cdef channel ch = <channel>data    ch._sock_state_callback(s, read, write)

Gevent_sock_state_callback only calls the _ sock_state_callback of the channel and sets it to read or write.

Cdef _ sock_state_callback (self, int socket, int read, int write): If not self. channel: Return cdef object watcher = self. _ watchers. get (socket) cdef int events = 0 if read: Events | = ev_read if write: Events | = ev_write if watcher is none: If not events: Return watcher = self. loop. io (socket, events) # For the first time, start Io watcher self. _ watchers [socket] = watcher Elif events: # Check whether the event has changed if watcher. events = events: Return watcher. stop () watcher. events = events # set the new status else: Watcher. stop () self. _ watchers. pop (socket, none) if not self. _ watchers: Self. _ timer. stop () return # if there is no event, that is, all the operations are completed, Watcher will call back our final callback function (for example, the callback set when gethostbyname is called. start (self. _ process_fd, watcher, pass_events = true) # watcher sets the callback self. _ timer. again (self. _ on_timer) # Let C-Ares process the timeout and broken_connections every second.

Io wather callback self. _ process_fd is mainly to call cares. ares_process_fd to continue processing the specified file descriptor,
Cares. ares_socket_bad indicates that the event is not processed, that is, the event has been processed.

Def _ process_fd (self, int events, object watcher): If not self. channel: Return cdef int read_fd = watcher. FD # file descriptor cdef int write_fd = read_fd if not (events & ev_read): # if there is no readable event, set read FD to "not processed" read_fd = cares. ares_socket_bad if not (events & ev_write): # No writeable event. Set the write FD to "not processed" write_fd = cares. ares_socket_bad cares. ares_process_fd (self. channel, read_fd, write_fd)

In fact, the above C-Ares process is almost the same, and the final callback will be called back. Let's take a look at the gethostbyname operation.
The gethostbyname function defined in resolver_ares.py calls gethostbyname_ex.

Def gethostbyname_ex (self, hostname, family = af_inet): While true: Ares = self. ares try: Waiter = waiter (self. hub) # use waiter Ares. gethostbyname (Waiter, hostname, family) # call ARES. gethostbyname, set the callback to waiter result = waiter. get () # We know that when waiter has no results, it will switch to the hub. It works perfectly with gevent. If not result [-1]: Raise gaierror (-5, 'No address associated with hostname') return result failed t gaierror: If Ares is self. ares: Raise

Waiter defines the _ call _ method, so it can be directly used as a callback function.
Ares. gethostbyname mainly calls cares. ares_gethostbyname (self. Channel, name, family, <void *> gevent_ares_host_callback, <void *> Arg)
Gevent_ares_host_callback is called back when the DNS request is successful or fails.
Gevent_ares_host_callback calls back the above waiter and sends the result to waiter. You can check the code here, which is relatively simple.
Waiter. _ call _ switches to the previously switched greenlet, that is, the previous waiter. Get (), and returns result. gethostbyname is successfully executed.

C-Ares is really beautiful. By providing just a few interfaces, You can perfectly combine yourself with other frameworks. Very nice !!!

I was curious about the running mode of C-Ares before, and the details of internal DNS may not be concerned. The focus is to combine the problem and spent two nights writing this article.

The article is very valuable.

[Gevent source code analysis] C-Ares asynchronous DNS requests