Uses python and libnmapd to extract Nmap scan results

This article describes how to use a line of code to parse nmap scan results. the NmapParser Library in libnmap is used in the Python environment, this library can easily help us parse nmap scan results. Whenever I perform intranet penetration for a large number of hosts and services, I am always used to extracting information from nmap scan results in an automated manner. This helps to automatically detect different types of services, such as Path cracking for web services, testing keys or protocols used by SSL/TLS services, and other targeted tests.

I often use IPthon or * nix shell in penetration tests, which can be accessed through Python, whether directly used in scripts or in REPL environments, or write the code to the disk and then access it through shell commands.

Configuration
The first step to parse the nmap scan result is to perform an nmap scan. I don't want to focus on too many details here, but if you want to directly use the code in this article, you need to save the scan structure to an xml file (-oX or-oA) the service detection (-sV) and running related scripts (-SC) are executed on the open port ).
The command in this article assumes that you are running in a Python REPL environment such as IPython and installing the libnmap module (which can be installed using easy_install or pip.
Before starting, you need to set the corresponding environment. first, import the NmapParser module and read your xml scan result file (the instance name "up_hosts_all_ports_fullscan.xml" is located in the current working directory)

from libnmap.parser import NmapParsernmap_report = NmapParser.parse_fromfile('up_hosts_all_ports_fullscan.xml')

The rest of this article contains a series of useful information extracted using a single line of code. All the examples assume that the nmap scan results are saved in a file as shown above. The following provides some basic sample code. if you want to run them directly in IPython, run the above code first so that it can be output directly on the console for your convenience. I usually do this step first, so that I can ensure that the output data is the same as expected.
Then, you can select a variable name and assign the data to the variable using "=", so that you can directly call it in the subsequent code, or write it to the disk for shell command use. If you want to use some things for multiple times, you can paste some code segments into the Python script, or want to add some more complex logic, but this may make the REPL environment hard to handle, in the last section, I will describe how to perform these operations quickly.

Port information
Host that opens the specified port number
Displays all hosts with the specified port number enabled. Generate a list containing host address (string. The following uses port 443 as an example. you can change it to the value you need.

 [ a.address for a in nmap_report.hosts if (a.get_open_ports()) and 443 in [b[0] for b in a.get_open_ports()] ]

Number of open ports

Displays the number of open ports on a series of hosts. Generate a list containing port count (int) and sort it.

 sorted(set([ b[0] for a in nmap_report.hosts for b in a.get_open_ports()]), key=int)

Services corresponding to open ports on the host, grouped by port number

The port numbers open to all hosts are displayed, grouped and sorted by port numbers. Generate a list containing multiple lists (each element in the list is also a list). The first element in each member list is the port number (int ), the second element is a list containing the IP address (string) of the host that opens the corresponding port.

[[A, [B. address for B in nmap_report.hosts for c in B. get_open_ports () if a = c [0] for a in sorted (set ([B [0] for a in nmap_report.hosts for B in. get_open_ports ()]), key = int)] SSL/TLS and HTTP/HTTPS

Hosts and ports using SSL

Displays all hosts and ports that use SSL. This is done by checking whether a service has used the "SSL" channel or related script detection results that contain the pem certificate. Generate a list containing a series of lists. each member list contains the host address (string) and port number (int ).

 [ [a.address, b.port] for a in nmap_report.hosts for b in a.services if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results) ]

The following content contains the same information, but not a list containing the list. Instead, the join function is used to create a list containing "Host: port number" (string.

 [ ':'.join([a.address, str(b.port)]) for a in nmap_report.hosts for b in a.services if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results) ]

Hosts and ports that contain web services

Display all web services and their corresponding port numbers and protocols (http or https ). This generates a list containing multiple lists. each member list contains the protocol (string), address (string), and port number (int ). However, there may be some problems here. When nmap reports a website using https, it sometimes shows that the service is "https ", sometimes it is displayed as "http" using the "ssl" channel, so I adjusted the data format for unified output.

 [ [(b.service + b.tunnel).replace('sl',''), a.address, b.port] for a in nmap_report.hosts for b in a.services if b.open() and b.service.startswith('http') ]

The url (string) is added to the list of protocols, hosts, and port numbers ).

 [ (b.service + b.tunnel).replace('sl','') + '://' + a.address + ':' + str(b.port) + '/' for a in nmap_report.hosts for b in a.services if b.open() and b.service.startswith('http') ]

Other service information
Unknown service

Displays all services that nmap cannot recognize. Generate a list containing multiple lists. each member list contains the address (string), port number (int), and port fingerprint (string) scanned by nmap ). This information is generated primarily to facilitate subsequent manual review of specific services without being involved in any automated process.

 [ [ a.address, b.port, b.servicefp ] for a in nmap_report.hosts for b in a.services if (b.service =='unknown' or b.servicefp) and b.port in [c[0] for c in a.get_open_ports()] ]

Software Recognized by nmap
Displays all the software identified in the nmap scan. Generate a list sorted by product letters.

 sorted(set([ b.banner for a in nmap_report.hosts for b in a.services if 'product' in b.banner]))

Host and port number corresponding to the software, grouped by product
The host and Port corresponding to the software are displayed and grouped by product. Generate a list containing multiple lists. The first element of each member list is the software name (string), followed by the address (string) and port number (int ).

 [ [ a, [ [b.address, c.port] for b in nmap_report.hosts for c in b.services if c.banner==a] ] for a in sorted(set([ b.banner for a in nmap_report.hosts for b in a.services if 'product' in b.banner])) ]

The same information as above, but the output is slightly different. A list containing multiple lists is also generated. The first element of the member list is the software name (string), but the second is a list containing "Host: port number.

 [ [ a, [ ':'.join([b.address, str(c.port)]) for b in nmap_report.hosts for c in b.services if c.banner==a] ] for a in sorted(set([ b.banner for a in nmap_report.hosts for b in a.services if 'product' in b.banner])) ]

Search for hosts and ports related to a specified keyword
Displays all hosts and ports associated with the given keywords, and searches for the product name and service name from the original text of the nmap scan results. The following uses Oracle as an example. Generate a list containing multiple lists. each member list contains the host address (string) and port number (int ).

 [ [a.address, b.port] for a in nmap_report.hosts for b in a.services if b.open() and 'Oracle' in str(b.get_dict()) + str(b.scripts_results) ]

The same method is used, except that all stored information is searched in lower case after being modified (in the following example, the lower case "oracle"). the output format is the same as above.

 [ [a.address, b.port] for a in nmap_report.hosts for b in a.services if b.open() and 'oracle' in (str(b.get_dict()) + str(b.scripts_results)).lower() ]

Other things

Same certificate name
The SSL certificate found is the same as the certificate name obtained after being parsed using the nmap script. In this way, when you start scanning from an IP address and reverse DNS becomes invalid, you can help determine the host name of the system. Generate a list containing multiple lists. each member list contains the IP address (string) and extracted host name (string ).

 [ [a.address, c['elements']['subject']['commonName'] ] for a in nmap_report.hosts for b in a.services for c in b.scripts_results if c.has_key('elements') and c['elements'].has_key('subject') ]

Method for processing the above results

As mentioned above, when you paste it directly into IPython REPL, the output is printed on the screen. This is really good, because you can view the information you are interested in at any time, but you may want to do more. One of the major advantages of generating the above information is that you can easily perform some automated operations based on the results.
If you are familiar with Python and should be able to do this easily, you can skip this section. But if you are not familiar with it, this section describes some basic knowledge and shows you how to use the above code snippet.

Save to disk
If you want to save the output result of the above code segment to a text file on the disk, you need to convert the output list to an appropriate string format (depending on your needs ), then, write the string to the file. In Python, you can use the join function to integrate these lists and write them into files. here is just an example.
We want to extract the hosts and ports that support SSL from the generated list and save them to a new file, in this way, you can use a loop in bash to complete and use the command line tool for testing.
I usually use a line of code in IPython to do this. although a line of code is more convenient, I will split the code here for ease of reading and understanding.
Let's generate a list containing "Host: Port" before parsing. Please note that we use the str function to replace the port number from the integer type into the character type, in this way, the join function can be used to splice other strings.

 [ ':'.join([a.address, str(b.port)]) for a in nmap_report.hosts for b in a.services if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results) ]

Let's assign the variable "ssl_services" to the result of the above code to facilitate subsequent calls.

 ssl_services = [ ':'.join([a.address, str(b.port)]) for a in nmap_report.hosts for b in a.services if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results) ]

Now, let's use the join function to splice every element in the list and use ('\ n') for line feed. then assign it a variable named "ssl_services_text.

 ssl_services_text = '\n'.join(ssl_services)

Then, we can create a new file named “ssl_services_file.txt in the current working directory and write the content of the variable "ssl_services_text.

Open('ssl_services_file.txt ', 'w'). write (ssl_services_text)

This is simple. you can use the file content as needed.

Use other Python code
Maybe you want to use other Python code to complete the above work? It is also very simple. The following is another example. here we traverse the web service identified by each nmap and the request results of its webpage.
The following will generate a list containing URLs. we will assign a variable named "urls" to it.

 urls = [ (b.service + b.tunnel).replace('sl','') + '://' + a.address + ':' + str(b.port) + '/' for a in nmap_report.hosts for b in a.services if b.open() and b.service.startswith('http') ]

Next, we will perform some preparation work first, import the requests module, set a simple getAndSave function for web requests, and save the returned results to the disk. the file name is automatically generated by url. You may notice that the "verify = False" option is used in the following code in the get request. this will ignore the certificate verification error when sending the request, this option is often used when testing an internal machine because the internal machine basically does not have an SSL certificate issued by a trusted certificate authority.

 import requestsdef getAndSave(url):r = requests.get(url, verify=False)open('_'.join(url.split('/')[2:]).replace(':',''),'wb').write(r.text.encode('utf8'))

Now, let's add some code to traverse every url, request the robots.txt file of each site, and save it locally for future use.

 for a in urls:getAndSave(a + 'robots.txt')

In this way, the robots.txt file of each site is crawled to the current working directory. This is just a simple example.

Summary

I hope that after reading this article, you can use Python to parse nmap scan results flexibly.

For more information about how to use python and libnmapd to extract Nmap scan results, see PHP!