The OpenStack Swift container server process often inexplicably enters D DS and other states
Record this time. Storage.error Log for easy analysis
One scenario is that the following log
June316:46:host207 container-server:error __call__ ERROR with put/sda5/100256/AUTH_3ED1D3F47B4E4794A59B7217BF136EEE/ATEST1/DATA4M: #012Traceback (most recent): #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line498,In __call__#012 Res = Method (req) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line2217,In wrapped#012return func (*a, **kw) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line837,In _timing_stats#012 RESP = func (ctrl, *args, **kwargs) #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line271,In put#012 broker.put_object (obj, timestamp,Int (req.headers[‘X-size‘]),#012 File"/usr/lib/python2.7/site-packages/swift/common/swob.py", line219,In __getitem__#012return Self.environ[self._normalize (Key)] #012KeyError:‘Http_x_size' (txn:tx1b48300503fc454d82e07-0055498A24) June316:51:host207 container-server:error __call__ ERROR with put/sda5/100256/AUTH_3ED1D3F47B4E4794A59B7217BF136EEE/ATEST1/DATA4M: #012Traceback (most recent): #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line498,In __call__#012 Res = Method (req) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line2217,In wrapped#012return func (*a, **kw) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line837,In _timing_stats#012 RESP = func (ctrl, *args, **kwargs) #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line271,In put#012 broker.put_object (obj, timestamp,Int (req.headers[‘X-size‘]),#012 File"/usr/lib/python2.7/site-packages/swift/common/swob.py", line219,In __getitem__#012return Self.environ[self._normalize (Key)] #012KeyError:‘Http_x_size' (txn:tx1b48300503fc454d82e07-0055498A24) June316:56:host207 container-server:error __call__ ERROR with put/sda5/100256/AUTH_3ED1D3F47B4E4794A59B7217BF136EEE/ATEST1/DATA4M: #012Traceback (most recent): #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line498,In __call__#012 Res = Method (req) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line2217,In wrapped#012return func (*a, **kw) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line837,In _timing_stats#012 RESP = func (ctrl, *args, **kwargs) #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line271,In put#012 broker.put_object (obj, timestamp,Int (req.headers[‘X-size‘]),#012 File"/usr/lib/python2.7/site-packages/swift/common/swob.py", line219,In __getitem__#012return Self.environ[self._normalize (Key)] #012KeyError:‘Http_x_size' (txn:tx1b48300503fc454d82e07-0055498A24) June317:01:host207 container-server:error __call__ ERROR with put/sda5/100256/AUTH_3ED1D3F47B4E4794A59B7217BF136EEE/ATEST1/DATA4M: #012Traceback (most recent): #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line498,In __call__#012 Res = Method (req) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line2217,In wrapped#012return func (*a, **kw) #012 File"/usr/lib/python2.7/site-packages/swift/common/utils.py", line837,In _timing_stats#012 RESP = func (ctrl, *args, **kwargs) #012 File"/usr/lib/python2.7/site-packages/swift/container/server.py", line271,In put#012 broker.put_object (obj, timestamp,int (Req.headers[ X-size012 File /usr/lib/python2.7/site-packages/swift/common/swob.py219, in __getitem__#012 return self.environ[self._normalize (Key)] #012KeyError: ' http_x_size "(TXN: TX1B48300503FC454D82E07-0055498A24) June 3 17:59:27 host207 Object-replicator:lockup detected. Killing Live Coros. June 3 18:23:19 host207 container-server:sigterm Receivedjun 3 18:23:19 host207 account-server:sigterm Receivedjun 3 18:2 3:19 host207 Object-server:sigterm received
Swift container server inexplicable stuck