詳解python並發擷取snmp資訊及效能測試方法

本篇文章主要介紹了詳解python並發擷取snmp資訊及效能測試，小編覺得挺不錯的，現在分享給大家，也給大家做個參考。一起跟隨小編過來看看吧

python & snmp

用python擷取snmp資訊有多個現成的庫可以使用，其中比較常用的是netsnmp和pysnmp兩個庫。網上有較多的關於兩個庫的例子。

本文重點在於如何並發的擷取snmp的資料，即同時擷取多台機器的snmp資訊。

netsnmp

先說netsnmp。python的netsnmp，其實是來自於net-snmp包。

python通過一個c檔案調用net-snmp的介面擷取資料。

因此，在並發擷取多台機器的時候，不能夠使用協程擷取。因為使用協程，在get資料的時候，協程會一直等待net-snmp介面返回資料，而不會像socket使用時那樣在等待資料時把CPU切換給其他協程使用。從這點上來說，使用協程和串列擷取沒有區別。

那麼如何解決並發擷取的問題呢？可以使用線程，多線程擷取(當然也可以使用多進程)。多個線程同時調用net-snmp的介面擷取資料，然後cpu在多個線程之間不停切換。當一個線程擷取一個結果後，可以繼續調用介面擷取下一個snmp資料。

這裡我寫了一個範例程式。首先把所有的host和oid做成任務放到隊列裡，然後啟動多個線程，去執行擷取任務。程式範例如下：

import threadingimport timeimport netsnmpimport Queuestart_time = time.time()hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114",     "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120",     "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15",     "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136",     "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140",     "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127",     "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118",     "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62",     "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31",     "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35",     "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133",     "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155",     "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231",     "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14",     "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2",     "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232",     "192.24.212.231", "192.24.212.230"]oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",    ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",    ".1.3.6.1.4.1.2021.4.15.0"]myq = Queue.Queue()rq = Queue.Queue()#把host和oid組成任務for host in hosts:  for oid in oids:    myq.put((host,oid))def poll_one_host():  while True:    try:      #死迴圈從隊列中擷取任務，直到隊列任務為空白      host, oid = myq.get(block=False)      session = netsnmp.Session(Version=2, DestHost=host, Community="cluster",Timeout=3000000,Retries=0)      var_list = netsnmp.VarList()      var_list.append(netsnmp.Varbind(oid))      ret = session.get(var_list)      rq.put((host, oid, ret, (time.time() - start_time)))    except Queue.Empty:      breakthread_arr = []#開啟多線程num_thread = 50for i in range(num_thread):  t = threading.Thread(target=poll_one_host, kwargs={})  t.setDaemon(True)  t.start()  thread_arr.append(t)#等待任務執行完畢for i in range(num_thread):  thread_arr[i].join()while True:  try:    info = rq.get(block=False)    print info  except Queue.Empty:    print time.time() - start_time    break

netsnmp除了支援get操作之外，還支援walk操作，即遍曆某個oid。

但是walk使用的時候需要謹慎，以免導致高延時等問題，具體可以參見之前的一篇snmpwalk高延時問題分析的部落格。

pysnmp

pysnmp是用python實現的一套snmp協議的庫。其自身提供了對於非同步支援。

import timeimport Queuefrom pysnmp.hlapi.asyncore import *t = time.time()myq = Queue.Queue()#回呼函數。在有資料返回時觸發def cbFun(snmpEngine, sendRequestHandle, errorIndication, errorStatus, errorIndex, varBinds, cbCtx):   myq.put((time.time()-t, varBinds))hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114",     "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120",     "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15",     "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136",     "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140",     "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127",     "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118",     "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62",     "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31",     "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35",     "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133",     "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155",     "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231",     "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14",     "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2",     "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232",     "192.24.212.231", "192.24.212.230"]oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",    ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",    ".1.3.6.1.4.1.2021.4.15.0"]    snmpEngine = SnmpEngine()#新增工作for oid in oids:  for h in hosts:    getCmd(snmpEngine,      CommunityData('cluster'),      UdpTransportTarget((h, 161), timeout=3, retries=0,),      ContextData(),      ObjectType(ObjectIdentity(oid)),      cbFun=cbFun)time1 = time.time() - t#執行非同步擷取snmpsnmpEngine.transportDispatcher.runDispatcher()#列印結果while True:  try:    info = myq.get(block=False)    print info  except Queue.Empty:    print time1    print time.time() - t    break

pysnmp本身只支援最基礎的get和getnext命令，因此如果想使用walk，需要自己進行實現。

效能測試

在同一個環境下，對兩者進行了效能測試。兩者對198個host，10個oid進行採集。

測試組	耗時(sec)
netsnmp(20線程)	6.252
netsnmp(50線程)	3.269
netsnmp(200線程)	3.265
pysnmp	4.812

可以看到netsnmp的採集速度跟線程數有關。當線程數增大到一定程度，採集時間不再縮短。因為開闢線程同樣會消耗時間。而已有的線程已經足夠處理。

pysnmp效能較之略差一下。詳細分析pysnmp在新增工作(執行getCmd時)消耗了約1.2s，之後的採集約消耗3.3秒。

在增加了oid數，在進行實驗。host仍然是198個，oid是42個。

測試組	耗時(sec)
netsnmp(20線程)	30.935
netsnmp(50線程)	12.914
netsnmp(200線程)	4.044
pysnmp	11.043

可以看到差距被進一步拉大。線上程足夠多的情況下，netsnmp的效率要明顯強於pysnmp。

因為二者都支援可以並行採集多個host，從易用性來說，netsnmp更為簡單一些，且netsnmp支援walk功能。本文更加推薦netsnmp。

安裝netsnmp需要安裝net-snmp。如果centos，則使用yum會較為方便。