Swift Open STATSD After uploading data, there is a return 503 of the bug

Reprint Please specify source: http://blog.csdn.net/cywosp/article/details/40781569

Swift before the version number 2.1.0 assumes that the configuration file for each service opens below configuration, and the system is not configured correctly there will be an error in uploading objects

log_statsd_host = localhost log_statsd_port = 8125 log_statsd_default_sample_rate = 1.0 Log_statsd_sample_rate_factor = 1.0 Log_statsd_metric_prefix =

Detailed error log information is probably for example the following:Object-server error __call__ error with put/sdc/2468/auth_8f9dbbadd64a43a0abb5e832c6ea766a/000008/013781: # 012Traceback (most recent): #012  file "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 938, In __call__#012    res = Method (req) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py ", line 1558, in wrapped#012    return func (*a, **kw) #012  file"/usr/lib/python2.6/site-packages/swift/ common/utils.py ", line 520, in _timing_stats#012    RESP = func (ctrl, *args, **kwargs) #012  file"/usr/lib/ python2.6/site-packages/swift/obj/server.py ", line 712, in put#012    file.put (FD, metadata) #012  file" /usr/lib64/python2.6/contextlib.py ", line-in __exit__#012    self.gen.throw (type, value, Traceback) #012 & nbsp File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 286, in mkstemp#012    yield fd#012   File "/usr/lib/python2.6/site-packages/swift/obj/server.py ", line 680, in put#012     ' PUT ' + device + '. Timing ', Elapsed_time, upload_size) #012  file"/usr/lib /python2.6/site-packages/swift/common/utils.py ", line 654, in wrapped#012    return func (SELF.LOGGER.STATSD _client, *a, **kw) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 506, in Transfer_rate #012    sample_rate) #012  file "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 496, in timing#012    return self._send (Metric, Timing_ms, ' Ms ', sample_rate) #012  file "/usr/lib/python2.6/ site-packages/swift/common/utils.py ", line 481, in _send#012    return sock.sendto (' | '). Join (Parts), self._target) #012  file "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 371, in SendTo #012    return self.fd.sendto (*args) #012error: [Errno 1]Operation not permitted(txn:tx8d76698250304466817aa99061637421)

according to the log information is found in the swift/common/utils.py file Statsdclient._send function throws an exception is not captured caused. The function code is as follows:def _send (self, m_name, m_value, M_type, sample_rate):if Sample_rate is None:sample_rate = self._default_sample_ratesample_rate = sample_rate * Self._sample_rate_factorparts = ['%s%s:%s '% (Self._prefix, M_name, M_value), M_type]If sample_rate < 1:if Self.random () < sample_rate:parts.append (' @%s '% (sample_rate,))Else:return# Ideally, we ' d cache a sending socket in# results in a sockets getting shared by multiple green threads.With closing (Self._open_socket ()) as sock:return sock.sendto (' | '). Join (Parts), self._target) #该函数调用抛出了异常 The workaround:in return sock.sendto (' | '). Join (Parts), self._target) to increase the exception handling. Detailed code to refer to the official latest code
at the same time, a large number of/var/log/messages logs appear in the system such as the following information:

proxy-access Error sending UDP message to (' 127.0.0.1 ', 8125): [Errno 1] Operation not permitted PROXY-ACC ESS Error sending UDP message to (' 127.0.0.1 ', 8125): [Errno 1] Operation not permitted kernel: __ratelimit:89 CA Llbacks suppressed kernel:nf_conntrack:table full, dropping packet. kernel:nf_conntrack:table full, dropping packet. kernel:nf_conntrack:table full, dropping packet.

The workaround:

The kernel reported a packet drop error at the same time. The main reason is that the server firewall is causing the filtering mechanism (Net.ipv4.netfilter.ip_conntrack_max is too small). You can shut down the firewall here (Service iptables stop)

References: [1] https://bugs.launchpad.net/swift/+bug/1183152 [2] http://www.cyberciti.biz/faq/ip_ conntrack-table-ful-dropping-packet-error/ [3] http:// STACKOVERFLOW.COM/QUESTIONS/6240951/SENDTO-OPERATION-NOT-PERMITTED-NETSNMP

copyright notice: This article Bo Master original article. Blog, not reproduced without consent.

The presence of the uploaded data after Swift's open STATSD, with a bug in return 503