Details about how to obtain snmp information and performance tests concurrently in python, pythonsnmp

Details about how to obtain snmp information and performance tests concurrently in python, pythonsnmp

Python & snmp

Multiple existing libraries can be used to obtain snmp information using python.netsnmpAndpysnmpTwo databases. There are many examples of two databases on the Internet.

This article focuses on how to concurrently obtain snmp data, that is, to obtain snmp information from multiple machines at the same time.

Netsnmp

Let's talk about netsnmp first. Python's netsnmp actually comes from the net-snmp package.

Python uses a c file to call the net-snmp interface to obtain data.

Therefore, you cannot use coroutine to obtain multiple machines concurrently. Because the coroutine uses the coroutine, The coroutine waits for the net-snmp interface to return data during the get data process, instead of switching the CPU to other coroutine when waiting for data as the socket is used. From this point, there is no difference between coroutine and serial get.

So how can we solve the problem of concurrent access? You can use threads and multiple threads to obtain them (of course, you can also use multiple processes ). Multiple Threads call the net-snmp interface to obtain data at the same time, and the cpu continuously switches between multiple threads. After a thread obtains a result, it can continue to call the interface to obtain the next snmp data.

Here I wrote a sample program. First, all the host and oid tasks are put into the queue, and then multiple threads are started to execute the acquisition task. The program sample is as follows:

Import threadingimport timeimport netsnmpimport Queuestart_time = time. time () hosts = ["192.0000150.109", "192.0000150.110", "192.0000150.111", "expires", "192.0000150.114", "192.0000150.115", "expires ", "192.0000150.117", "medium", "medium", "192.0000150.120", "medium", "192.000080.149", "192.000096.59", "192.000082.14", "192.000082.15 ", "192.000082.17", "192.000082.19", "192.000082.12", "192.000080.large", "medium", "192.000080.large", "medium", "192.000080.130", "192.000080.130 ", "plugin", "192.000081.140", "192.000082.26", "192.000082.28", "192.000082.23", "192.000082.21", "192.000080.128", "plugin", "plugin", "average ", "region", "192.000080.118", "192.000080.119", "region", "192.000080.116", "192.000080.115", "192.000078.62 ", "classification", "192.000081.125", "classification", "classification", "192.000082.33", "192.000082.32", "192.000082.30", "192.000081.128", "192.000082.39 ", "classification", "192.20.82.35", "192.20.81.130", "192.20.80.200", "192.20.81.20."," classification "," 192.20.81.20.", "classification", "192.20.82.43 ", "192.20.82.45", "192.20.82.41", "medium", "192.20.79.155", "192.20.79.154", "medium", "192.25.76.large", "medium ", "clerk", "192.25.000096", "192.25.000095", "192.25.000094", "192.25.000093", "clerk", "192.24.163.29", "clerk", "clerk ", "Expiration", "192.24.193.2", "192.24.193.19", "192.24.193.18", "192.18.157.132", "192.18.157.20."," 192.24.212.20.", "Expiration ", "192.24.212.230"] oids = [". 1.3.6.1.4.1.2021.11.9.0 ",". 1.3.6.1.4.1.2021.11.10.0 ",". 1.3.6.1.4.1.2021.11.11.0 ",". 1.3.6.1.4.1.2021.10.1.3.1 ",". 1.3.6.1.4.1.2021.10.1.3.2 ",". 1.3.6.1.4.1.2021.10.1.3.3 ",". 1.3.6.1.4.1.2021.4.6.0 ",". 1.3.6.1.4.1.2021.4.14.0 ",". 1.3.6.1.4.1.2021.4.15.0 "] myq = Queue. queue () rq = Queue. queue () # Make the host and oid a task for host in hosts: for oid in oids: myq. put (host, oid) def poll_one_host (): while True: try: # obtain tasks from the queue through an endless loop until the queue task is empty host, oid = myq. get (block = False) session = netsnmp. session (Version = 2, DestHost = host, Community = "cluster", Timeout = 3000000, Retries = 0) var_list = netsnmp. varList () var_list.append (netsnmp. varbind (oid) ret = session. get (var_list) rq. put (host, oid, ret, (time. time ()-start_time) wait t Queue. empty: breakthread_arr = [] # enable multithreading num_thread = 50for I in range (num_thread): t = threading. thread (target = poll_one_host, kwargs ={}) t. setDaemon (True) t. start () thread_arr.append (t) # Wait until the task is finished. for I in range (num_thread): thread_arr [I]. join () while True: try: info = rq. get (block = False) print info failed t Queue. empty: print time. time ()-start_time break

In addition to the get operation, netsnmp also supports the walk operation, that is, traversing a oid.

However, you need to be cautious when using the walk to avoid problems such as high latency. For details, see the previous snmpwalk high latency analysis blog.

Pysnmp

Pysnmp is a set of snmp protocol libraries implemented in python. It provides asynchronous support.

Import timeimport Queuefrom pysnmp. hlapi. asyncore import * t = time. time () myq = Queue. Queue () # callback function. When data is returned, def cbFun (snmpEngine, sendRequestHandle, errorIndication, errorStatus, errorIndex, varBinds, cbCtx) is triggered: myq. put (time. time ()-t, varBinds) hosts = ["192.0000150.109", "192.0000150.110", "expires", "192.0000150.113", "192.0000150.114", "expires ", "clerk", "192.0000150.117", "clerk", "clerk", "192.0000150.120", "clerk", "192.000080.149", "192.000096.59", "192.000082.14 ", "192.000082.15", "192.000082.17", "192.000082.19", "192.000082.12", "192.000080.133", "medium", "192.000080.htm", "medium ", "192.20.80.130", "2017"," 192.20.81.140 "," 192.20.82.26 "," 192.20.82.28 "," 192.20.82.21 "," 192.11680.128 "," renew "," renew ", "region", "region", "192.20.80.124", "192.20.81.151", "192.20.80.118", "region", "192.20.80.112", "192.20.80.116", "region ", "classification", "classification", "192.000081.125", "classification", "192.000082.33", "192.000082.31", "192.000082.32", "192.000082.30", "192.000081.128 ", "telephone", "192.20.82.37", "192.20.82.35", "192.20.81.130", "192.20.80.200", "telephone", "192.20.81.133", "telephone", "192.20.81.133", "telephone ", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155", "medium ", "clerk", "clerk", "192.25.000096", "192.25.000095", "192.25.000094", "192.25.000093", "clerk", "192.24.163.21", "clerk", "clerk ", "region", "region", "192.24.193.2", "192.24.193.19", "region", "192.18.157.132", "192.18.157.20."," 192.24.212.20.", "region ", "192.24.212.230"] oids = [". 1.3.6.1.4.1.2021.11.9.0 ",". 1.3.6.1.4.1.2021.11.10.0 ",". 1.3.6.1.4.1.2021.11.11.0 ",". 1.3.6.1.4.1.2021.10.1.3.1 ",". 1.3.6.1.4.1.2021.10.1.3.2 ",". 1.3.6.1.4.1.2021.10.1.3.3 ",". 1.3.6.1.4.1.2021.4.6.0 ",". 1.3.6.1.4.1.2021.4.14.0 ",". 1.3.6.1.4.1.2021.4.15.0 "] snmpEngine = SnmpEngine () # Add task for oid in oids: for h in hosts: getCmd (snmpEngine, CommunityData ('cluster'), UdpTransportTarget (h, 161 ), timeout = 3, retries = 0,), ContextData (), ObjectType (ObjectIdentity (oid), cbFun = cbFun) time1 = time. time ()-t # obtain snmpsnmpEngine asynchronously. transportDispatcher. runDispatcher () # print the result while True: try: info = myq. get (block = False) print info failed t Queue. empty: print time1 print time. time ()-t break

Pysnmp only supports the most basic get and getnext commands. Therefore, if you want to use walk, you must implement it yourself.

Performance Testing

In the same environment, the performance of the two is tested. The two collect 198 hosts and 10 oid.

Test Group	Time consumed (sec)
Netsnmp (20 threads)	6.252
Netsnmp (50 threads)	3.269
Netsnmp (thread 200)	3.265
Pysnmp	4.812

We can see that the collection speed of netsnmp is related to the number of threads. When the number of threads increases to a certain extent, the collection time will not be shortened. Because opening a thread also consumes time. Some threads are sufficient.

Pysnmp performance is slightly inferior. The detailed analysis shows that pysnmp consumes about 1.2 s when adding tasks (when getCmd is executed), and the subsequent collection consumes about 3.3 seconds.

When the number of oid is increased, the experiment is ongoing. There are still 198 hosts and 42 oid hosts.

Test Group	Time consumed (sec)
Netsnmp (20 threads)	30.935
Netsnmp (50 threads)	12.914
Netsnmp (thread 200)	4.044
Pysnmp	11.043

We can see that the gap is further widened. When there are enough threads, netsnmp is more efficient than pysnmp.

Because both support parallel collection of multiple hosts, netsnmp is easier to use and netsnmp supports the walk function. Netsnmp is recommended in this article.

To install netsnmp, you must install net-snmp. If centos is used, yum is more convenient.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.