Considerations for using Libcurl

1. Libcurl and Close_wait
Reproduced from: http://blog.sunshow.net/2010/03/libcurl-and-close-wait/

Call Libcurl download, and then use Netstat to see that a large number of TCP connections remain in the close_wait state
View the documentation description for Libcurl, with an option:

Curlopt_forbid_reuse

Pass a long. Set to 1 to make the next transfer explicitly close the connection was done. Normally, Libcurl keeps all connections alive and one transfer in case a succeeding one follows that can re-use them. This option should is used with caution and only if your understand what it does. Set to 0 to have Libcurl keep the connection open for possible later (default re-use).

That is, when Libcurl completes a task by default, the consideration of a reuse connection is not immediately closed
If there is no new TCP request to reuse this connection, then only wait until the close_wait timeout, which defaults to 7,200 seconds or higher, and too many close_wait connections can cause performance problems

Solution: curl_easy_setopt (Curl, curlopt_forbid_reuse, 1);

It's best to modify the TCP parameter to reduce the timeout for close_wait and time_wait

2. Libcurl Use Notes
Reproduced from: http://gcoder.blogbus.com/logs/54871550.html

Libcurl is a very good library, supporting HTTP,FTP and many other protocols. The most important thing about using a library is that it's dangerous to just look at an example and write a program without looking at the document carefully. My program crashed, I suspected that it was a problem with my code, and later found that it was useless to the library. Not looking at the document carefully (sometimes the document itself is also poor, in addition to look carefully, but also to move the brain, consider how it is achieved), the consequences are very serious. Without thinking about using someone else's library or code, sometimes it's cozy, but when something goes wrong, it's a sleepless.

1. Curlcode Curl_global_init (long flags); In multi-threaded applications, you need to call this function in the main thread. This function sets the desired environment for the libcurl. Typically, if you do not call it explicitly, the first time you call Curl_easy_init (), Curl_easy_init will invoke Curl_global_init, which is not a problem in a single-threaded environment. However, multithreading is not the case, because Curl_global_init is not thread-safe. Calling Curl_easy_int in multiple threads, and then if two threads simultaneously discover that Curl_global_init has not been invoked, and call Curl_global_init, the tragedy occurs. The probability of this happening is very small, but the possibility exists.

2. Libcurl has a very good feature, and it can even control the timeout of domain name resolution. However, by default, it is implemented using alarm + siglongjmp. Using alarm to do timeouts in multiple threads is almost impossible in itself. If you just use alarm, it doesn't cause the program to crash, but, plus siglongjmp, it's Killing You (the program crashes horribly, and there's almost no useful information in the core) because it requires a sigjmp_buf-type global variable and multithreading modifies it. (Typically, you can have a sigjmp_buf variable for each thread, in which case there is no problem using siglongjmp in multiple threads, but Libcurl has only one global variable and all threads will use it).

This is specifically similar to the timeout setting for curl_easy_setopt (Curl, Curlopt_timeout, 30L), resulting in the use of alarm (which is estimated to occur in the domain name resolution Phase), which is not possible in multiple threads, as mentioned earlier. The workaround is to disable alarm this timeout, curl_easy_setopt (curl, curlopt_nosignal, 1L).

This makes it safe to use timeouts in multiple threads. But the domain name resolution does not have the time-out mechanism, encounters very slow domain name resolution, also very troublesome. The recommended document is consider building Libcurl with C-ares support to enable asynchronous DNS lookups, which enables nice timeouts for n  ame resolves without signals. C-ares is an asynchronous DNS solution.

3. Libcurl Multithreading Usage considerations
Reproduced from: http://blog.csdn.net/jaylong35/article/details/6439549

1, the problem source, multithreading use Libcurl causes the program to run for a period of time after their exit, there is no obvious anomaly. The appropriate bug could not be found.

Finally, by looking at the data and some articles found on the Internet, it turns out that it is a signal processing problem:

curlopt_nosignal

Pass a long. If It is 1, Libcurl won't use any functions this install signal handlers or any functions this cause to be signals to the process. This option is mainly the Allow multi-threaded UNIX applications to still set/use all timeout options etc, without RIS King getting signals. (Added in 7.10)

If This option was set and Libcurl has been built with the standard name resolver, timeouts would not occur while the name R Esolve takes place. Consider building Libcurl with C-ares support to enable asynchronous DNS lookups, which enables nice timeouts for name Res Olves without signals.

Setting curlopt_nosignal to 1 makes Libcurl the "system to ignore sigpipe" signals, which otherwise are sent by the System when trying to send data to a socket which are closed in the other end. Libcurl makes a effort to never cause such sigpipes to trigger, but some-operating systems no have to way avoid and Even on those so have there are some corner when cases may they still, happen to our contrary.

is when multiple threads are using timeout processing, while the main thread has sleep or wait operation. If this option is not set, Libcurl will signal that the wait is interrupted to cause the program to exit.

So it's OK to set this option to 1 when you use it.

curl_setopt (Curl, curlopt_nosignal, 1L);

2, about the Libcurl library initialization and shutdown: Curl_global_init () and Curl_global_cleanup ()

These two functions are not thread safe. So you can only initialize and purge once in the main thread.

Although this is not necessarily a problem, but if not to deal with the probability of the occurrence.

problem: Multithreading Libcurl run for a period of time after the collapse, there is no definite point, there is no identified URL. Always look at the source code is not a problem, and finally through the debug trace is found in the access to SSL crash. http://blog.csdn.net/jaylong35/article/details/6988690

Just remember to OpenSSL is not supporting multithreading, to do their own lock processing. Furthermore, the associated lock operation is not supported in the Libcurl.

Solution:

Create a mutex function for OpenSSL when initializing the Libcurl, and a callback function to the OPENSS

OpenSSL lock L function prototype: void (* func) (int, int, const char *, int)

Set mode: Crypto_set_locking_callback (void (* func) (int, int, const char *, int));

Setting such a function is not enough, in addition to configure a lock ID callback function, this can refer to the use of OpenSSL multithreading related.

ID function prototype: unsigned int (*func) (void)

Set mode: crypto_set_id_callback (unsigned int (*func) (void));

These two settings allow you to resolve the problem of crash HTTPS multithreaded requests.

question: Curl_easy_perform has been blocked for a long time without returning,

#0 0xf7746430 in __kernel_vsyscall ()
#1 0x00580fd6 in poll () from/lib/libc.so.6
#2 0x081e77fa in Curl_socket_check ()
#3 0x081ff1ca in Curl_do_perform ()

WORKAROUND: Set timeout: curl_easy_setopt (curl, curlopt_timeout, 30L)