Background:
Online machines that need to filter access logs and send to another API
The beginning is a single process, too low efficiency, after the change to multi-process send, the view log occasionally abnormal error (forget ... )
Anyway, the port is not enough to use the error
Reason:
Each log is a request sent to the API, a short connection generates a large number of time_wait states, consumes a large number of ports
This high concurrency led to a large number of time_wait state kernel tuning is basically useless, and later instead of a long connection to solve the problem
The first version of the short connection version key code is as follows
Only the key parts of the code are posted because of the specific business information involved
Import pycurlwhere True: url=myqueue.get () Send_msg=pycurl. Curl () send_msg.setopt (Pycurl. Url,url) send_msg.perform () print send_msg.getinfo (send_msg. HTTP_CODE)
The modified post-long connection version is as follows:
Using the Requests Library
Import requestsclient=requests.session () headers = {' Content-type ': ' Application/json ', ' Connection ': ' Keep-alive '} where True: url=myqueue.get () r=client.get (url,headers=headers) print R.status_code