A bug that returned 503 when the upload data appeared after Swift opened STATSD

Reprint please indicate source: http://blog.csdn.net/cywosp/article/details/40781569

Swift before version 2.1.0 if the configuration file for each service has the following configuration open and the system is not configured correctly there will be an error in uploading the object

log_statsd_host = localhost log_statsd_port = 8125 log_statsd_default_sample_rate = 1.0 Log_statsd_sample_rate_factor = 1.0 Log_statsd_metric_prefix =

The specific error log information is probably as follows:Object-server error __call__ error with put/sdc/2468/auth_8f9dbbadd64a43a0abb5e832c6ea766a/000008/013781: # 012Traceback (most recent): #012  file "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 938, In __call__#012    res = Method (req) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py ", line 1558, in wrapped#012    return func (*a, **kw) #012  file"/usr/lib/python2.6/site-packages/swift/ common/utils.py ", line 520, in _timing_stats#012    RESP = func (ctrl, *args, **kwargs) #012  file"/usr/lib/ python2.6/site-packages/swift/obj/server.py ", line 712, in put#012    file.put (FD, metadata) #012  file" /usr/lib64/python2.6/contextlib.py ", line-in __exit__#012    self.gen.throw (type, value, Traceback) #012 & nbsp File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 286, in mkstemp#012    yield fd#012   File "/usr/lib/python2.6/site-packages/swift/obj/server.py ", line 680, in put#012     ' PUT ' + device + '. Timing ', Elapsed_time, upload_size) #012  file"/usr/lib /python2.6/site-packages/swift/common/utils.py ", line 654, in wrapped#012    return func (SELF.LOGGER.STATSD _client, *a, **kw) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 506, in Transfer_rate #012    sample_rate) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 496, in timing#012    return self._send (Metric, Timing_ms, ' Ms ', sample_rate) #012  file "/usr/lib/python2.6/ site-packages/swift/common/utils.py ", line 481, in _send#012    return sock.sendto (' | '). Join (Parts), self._target) #012  file "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 371, in SendTo #012    return self.fd.sendto (*args) #012error: [Errno 1]Operation not permitted(txn:tx8d76698250304466817aa99061637421)

according to the log information is found in the swift/common/utils.py file Statsdclient._send function throws an exception is not captured, the function code is as follows:def _send (self, m_name, m_value, M_type, sample_rate):if Sample_rate is None:sample_rate = self._default_sample_ratesample_rate = sample_rate * Self._sample_rate_factorparts = ['%s%s:%s '% (Self._prefix, M_name, M_value), M_type]If sample_rate < 1:if Self.random () < sample_rate:parts.append (' @%s '% (sample_rate,))Else:return# Ideally, we ' d cache a sending socket in# results in a sockets getting shared by multiple green threads.With closing (Self._open_socket ()) as sock:return sock.sendto (' | '). Join (Parts), self._target) #该函数调用抛出了异常 Workaround:in return sock.sendto (' | '). Join (Parts), self._target) to add exception handling, specific code can refer to the official latest code
at the same time, there are a lot of information in the/var/log/messages log of the system:

proxy-access Error sending UDP message to (' 127.0.0.1 ', 8125): [Errno 1] Operation not permitted PROXY-ACC ESS Error sending UDP message to (' 127.0.0.1 ', 8125): [Errno 1] Operation not permitted kernel: __ratelimit:89 CA Llbacks suppressed kernel:nf_conntrack:table full, dropping packet. kernel:nf_conntrack:table full, dropping packet. kernel:nf_conntrack:table full, dropping packet.

Workaround:based on the information above, it is known that Port 8125 is the STATSD service port, so there is a problem with the STATSD client. At the same time, the kernel reported a packet loss error, mainly because the server firewall turned on the filtering mechanism (Net.ipv4.netfilter.ip_conntrack_max too small), the firewall can be shut down (service iptables stop)

Reference:[1] https://bugs.launchpad.net/swift/+bug/1183152[2] http://www.cyberciti.biz/faq/ip_conntrack-table-ful-dropping-packet-error/[3] Http://stackoverflow.com/questions/6240951/sendto-operation-not-permitted-netsnmp

A bug that returned 503 when the upload data appeared after Swift opened STATSD