When StatsD is enabled in Swift, a 503 error is returned when uploading data.

When StatsD is enabled in Swift, a 503 error is returned when uploading data.

Before swift version 2.1.0, if the following configuration is enabled in the configuration file of each service and the system is not correctly configured, an error occurs when the upload object is incorrect.
Log_statsd_host = localhost
Log_statsd_ports = 8125
Log_statsd_default_sample_rate = 1.0
Log_statsd_sample_rate_factor = 1.0
Log_statsd_metric_prefix =
The specific error log information is as follows:
Object-server ERROR _ call _ error with PUT/sdc/2468/AUTH_8f9dbbadd64a43a0abb5e832c6ea766a/000008/013781: #012 Traceback (most recent call last ): #012 File "/usr/lib/python2.6/site-packages/swift/obj/server. py ", line 938, in _ call __# 012 res = method (req) #012 File"/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 1558, in wrapped #012 return func (* a, ** kw) #012 File "/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 520, in _ timing_stats #012 resp = func (ctrl, * args, ** kwargs) #012 File "/usr/lib/python2.6/site-packages/swift/obj/server. py ", line 712, in PUT #012 file. put (fd, metadata) #012 File "/usr/lib64/python2.6/contextlib. py ", line 34, in _ exit __# 012 self. gen. throw (type, value, traceback) #012 File "/usr/lib/python2.6/site-packages/swift/obj/server. py ", line 286, in mkstemp #012 yield fd #012 File"/usr/lib/python2.6/site-packages/swift/obj/server. py ", line 680, in PUT #012 'put. '+ device + '. timing', elapsed_time, upload_size) #012 File "/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 654, in wrapped #012 return func (self. logger. statsd_client, * a, ** kw) #012 File "/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 506, in transfer_rate #012 sample_rate) #012 File"/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 496, in timing #012 return self. _ send (metric, timing_ms, 'ms', sample_rate) #012 File "/usr/lib/python2.6/site-packages/swift/common/utils. py ", line 481, in _ send #012 return sock. sendto ('| '. join (parts), self. _ target) #012 File "/usr/lib/python2.6/site-packages/eventlet/greenio. py ", line 371, in sendto #012 return self. fd. sendto (* args) #012 error: [Errno 1] Operation not permitted (txn: tx8d76698250304466817aa99061637421)

According to the log information, the StatsdClient. _ send function in the swift/common/utils. py file throws an exception and is not caught. The function code is as follows:
Def _ send (self, m_name, m_value, m_type, sample_rate ):
If sample_rate is None:
Sample_rate = self. _ default_sample_rate
Sample_rate = sample_rate * self. _ sample_rate_factor
Parts = ['% s: % s' % (self. _ prefix, m_name, m_value), m_type]
If sample_rate <1:
If self. random () <sample_rate:
Parts. append ('@ % s' % (sample_rate ,))
Else:
Return
# Ideally, we 'd cache a sending socket in self, but that
# Results in a socket getting shared by multiple green threads.
With closing (self. _ open_socket () as sock:
Return sock. sendto ('|'. join (parts), self. _ target) # This function call throws an exception.
Solution:
Add exception handling to return sock. sendto ('|'. join (parts), self. _ target). For specific code, refer to the latest official code.

At the same time, the system's/var/log/messages log contains a large amount of information as follows:
Proxy-access Error sending UDP message to ('192. 0.0.1 ', 127): [Errno 1] Operation not permitted
Proxy-access Error sending UDP message to ('192. 0.0.1 ', 127): [Errno 1] Operation not permitted
Kernel: _ ratelimit: 89 callbacks suppressed
Kernel: nf_conntrack: table full, dropping packet.
Kernel: nf_conntrack: table full, dropping packet.
Kernel: nf_conntrack: table full, dropping packet.
Solution:
According to the above information, we know that port 8125 is the StatsD service port, so there is a problem with the StatsD client. At the same time, the kernel reports a packet loss error, mainly because the server firewall has enabled the filtering mechanism (net. ipv4.netfilter. ip_conntrack_max is too small). In this case, disable the firewall (service iptables stop)

Refer:
[1] https://bugs.launchpad.net/swift/+bug/1183152
[2] http://www.cyberciti.biz/faq/ip_conntrack-table-ful-dropping-packet-error/
[3] http://stackoverflow.com/questions/6240951/sendto-operation-not-permitted-netsnmp

Swift details: click here

This article permanently updates the link address: