Web penetration Security Testing

When conducting a security penetration test, we first need to collect as much information as possible for the target application. This task can be completed in different ways,
 
By using search engines, scanners, simple HTTP requests, or specially crafted requests, applications may leak information such as error information, version information, and technologies used.
 
One-stop test of the robots.txt File
 
Now, we will first introduce how to test the robots.txt file. Web Spider/robot/crawler can be used to search Web pages and further explore more and deeper Web content along the hyperlink. Of course, the website can store a robots.txt file in the root directory, so that it can specify which Web spider behaviors are acceptable to the site and those are prohibited.
 
For example, let's take a look at the internal volume disconnection of http://www.google.com/robots.txt:
 
User-agent: * Allow:/searchhistory/Disallow:/news? Output = xhtml & Allow:/news? Output = xhtmlDisallow:/searchDisallow:/groupsDisallow:/images...
 
The pseudo-command User-Agent indicates the specific Web spider/robot/Web crawler. For example, User-Agent: Googlebot indicates GoogleBot Web crawler, while User-Agent: * indicates all Web crawlers, robots, and Web Crawlers:
 
User-agent :*
 
The role of the pseudo-command Disallow is to specify which resources are disabled for spiders, robots, and web crawlers. In the preceding example, the spider is prohibited from accessing the following directories:
 
... Disallow:/searchDisallow:/groupsDisallow:/images...
 
Web Spider/robot/Web crawler can omit the "ban" in the robots.txt file ". Therefore, you do not need to use robots.txt as a panacea to restrict third-party access, storage, or post web content.
 
The following is a black box test and use case of the robots.txt file:
 
Wget
 
The Robots.txt file can be found in the Web root directory of the web server. For example, you can use wgetto retrieve robots.txt from www.google.comsite, as shown below:
 
$ Wget http://www.google.com/robots.txt--23:59:24--http://www.google.com/robots.txt=> 'robots.txt 'Resolving www.google.com... 74.125.19.103, 74.125.19.104, 74.125.19.147 ,... connecting to www.google.com | 74.125.19.103 |: 80... connected. HTTP request sent, awaiting response... 200 OKLength: unspecified [text/plain] [<=>] 3,425 --. -- K/s23: 59: 26 (13.67 MB/s)-'robots.txt 'saved [3425]
 
Use Google Webmaster toolsto analyze robots.txt
 
Google's Google Webmaster toolsimplements A robots.txtanalysis function. Therefore, we can use the function to analyze robots.txt during penetration testing. The specific method is as follows:
 
1. log on to Google Webmaster Tools with a Google account.
 
2. on the Dashboard, click the site URL you want to analyze.
 
3. Click Tools, and then click Analyze robots.txt.
 
 
 
 
 
2. Use search engines for Reconnaissance
 
The following describes how to search for Google Index and delete related web content from Google Cache. As we know, once the crawling process of GoogleBot is completed, it will be based on tags and related attributes (such
 
Bad request Your browser sent to query this server cocould not understand.
 
Response from SunONE 6.1:
 
$ Nc sunone.example.com 80 GET/JUNK/1.0Bad requestYour browser sent a query this server cocould not understand.
 
Automatic Test Method
 
There are multiple methods to obtain the Web server fingerprint. The manual method is described above. The following describes some automated testing methods through tools. Httprint is such a tool. Httprint has a Pattern Dictionary that identifies the type and version of the target server. Is a usage example:
 
 
Figure 3
 
Online Testing
 
Netcraft, an example of an online tool, can bring us a lot of useful information about the target server. Through it, we can retrieve information such as the operating system, the Web server used, the running time of the server, the Netblock owner, and the modification records related to the Web server and the operating system. For example:
 
 
Figure 4
 
V. Summary
 
When conducting a security penetration test, we first need to collect as much information as possible for the target application. This article introduces how to test the robots.txt file, use a search engine to collect useful information, and identify application portals. In the next part of this article, we will show you in detail how to test the applications running on the target address, and how to use the methods of getting useful messages in advance through error information.
 
 
When conducting a security penetration test, we first need to collect as much information as possible for the target application. This task can be completed in different ways,
 
By using search engines, scanners, simple HTTP requests, or specially crafted requests, applications may leak information such as error information, version information, and technologies used. This article describes in detail how to test which applications are running on the target address and how to use messages in advance through error messages.
 
1. Identify applications
 
When testing Web application vulnerabilities, the most important step is to find out which applications are hosted on the Web server. Many applications have known vulnerabilities and attack methods, allowing them to gain remote control or access confidential data. In addition, many applications often encounter configuration errors or are not updated for a long time, because some people always think that they are used internally, so they are ignored. In the past, the relationship between Web servers and IP addresses was usually one-to-one. However, with the rapid growth of virtual Web servers, many websites/applications share the same IP address.
 
As a security professional, sometimes a group of IP addresses must be processed to test a target server. The problem is that if the given IP address is an HTTP Service hosted on port 80, When you access the service by specifying an IP address, it reports that the address does not have messages such as Web Servers configured. In fact, the system may "hide" many Web applications, but they are given irrelevant Symbol names. Obviously, the breadth of analysis is greatly affected by the tested applications. You may not have noticed them, or just noticed some of them. Sometimes there are many target objects to be tested, such as a column of IP addresses and their corresponding symbol names. Even so, this list may only pass part of the information, that is, it may omit some symbolic names-because even customers do not know them, especially for those large organizations.
 
Other issues that affect the scope of audit are non-explicit Web applications that do not reference their URLs (such as http://www.example.com/some-strange-URL) from anywhere. This may be caused by incorrect configurations or intentional actions, such as non-public management interfaces. To solve this problem, web application testing is required.
 
The following describes the black box testing and examples. Web application detection is a process of searching for Web applications on a given infrastructure. These infrastructures are usually defined by a set of IP addresses, or a set of DNS Symbol names, or both. Either a typical penetration test or an application-centric evaluation test, this information needs to be provided before the actual audit. Unless otherwise specified in the employment contract (for example, "only test the application on http://www.example.com/"), audit should be conducted as much as possible, that is, it should find all applications accessible through a given target. In the following example, we will study some technologies that can achieve the above objectives.
 
Note: The following technologies apply to Internet-oriented Web servers, DNS, and Web-based reverse IP resolution services and search engines. In this example, we use a private IP address (such as 192.168.1.100) to represent a common IP address.
 
Three factors affect the number of applications related to a given DNS name (or an IP address:
 
1. Different base URLs
 
For a Web application, an obvious entry point is logging. In this example ". However, in general, we do not need to publish Web applications in this stealth mode unless you do not want to provide them in a standard way, but secretly inform your users of the specific location of these applications. However, this does not mean that these applications are hidden, but they are not published, but they are still there.
 
2. Non-Standard Port
 
Although Web applications are usually located at Port 80 (http) and port 443 (https), Web applications can be bound to any TCP port and referenced by specifying the port number, for example, http [s]: // www.example.com: port /. For example, For example, http://www.example.com: 20000 /.
 
3. VM
 
DNS allows us to map a single IP address to one or more Symbol names. For example, IP address 192.168.1.100 can be mapped to the following DNS names: names www.example.com, help;.example.com, and webmail.example.com. A vm generally uses this one-to-multiple method to provide different content. Specifies that the information of the VM we are referencing will be embedded in the Host: Header of HTTP 1.1.
 
Unless we know help;.example.com and webmail.example.com, we will not doubt that there are other Web applications.
 
The following describes how to solve these problems:
 
Solution to problem 1
 
In fact, we cannot thoroughly find all Web applications that use non-standard names. Because it is non-standard, there are no fixed standards to guide naming, but there are several techniques that penetration testers can use to gain some additional insights. First, if the Web server configuration is incorrect and you are allowed to browse the directory, you may find these applications. The security vulnerability scanner can also help us complete this task. Second, these applications can be referenced through other web pages. Therefore, they can be crawled by web crawlers or indexed by search engines. If we suspect that such a stealth application exists on www.example.com, we can use the site operator to search by google and then use "site: www.example.com" to check the search results. The returned URL may point to these non-explicit applications. Another way is to look at URLs that look like non-public applications. For example, a Web mail front-end can. For management interfaces (for example, a Tomcat Management Interface), you may hide the URL but it is not referenced in any other place. Therefore, it may be helpful to perform a dictionary search. The security vulnerability scanner is also useful in this regard.
 
Solution 2
 
It is relatively easy to check Web applications on non-standard ports. For example, you can use a port scanner, such as nmap, and the-sV option to identify services on any port, including the http [s] service. It is necessary to fully scan the 64 k tcp port address space. For example, the following command looks up all open ports whose IP address is 192.168.1.100 and tries to determine which services are bound to them:
 
Nmap-PN-sT-svs-p0-65535 192.168.1.100
 
Check the output and find the http or SSL encapsulated service flag. For example, the output result of the preceding command is as follows:
 
Interesting ports on 192.168.1.100 :( The 65527 ports scanned but not shown below are in state: closed) port state service VERSION22/tcp open ssh OpenSSH 3.5p1 (protocol 1.99) 80/tcp open http Apache httpd 2.0.40 (Red Hat Linux )) 443/tcp open ssl OpenSSL901/tcp open http Samba SWAT administration server1241/tcp open ssl Nessus security scanner3690/tcp open unknown8000/tcp open http-alt? 8080/tcp open http Apache Tomcat/Coyote JSP engine 1.1 from this example we can see:
 
◆ There is an Apache HTTP server license on port 80
 
◆ There seems to be an https server on port 443. Further confirmation is required. For example, you can use a browser to access https: // 192.168.1.100. Bytes
 
◆ A Samba SWAT Web interface is available on port 901. Bytes
 
◆ The service on port 1241 is not https, but the Nessus daemon process encapsulated by SSL. Bytes
 
◆ Port 3690 runs an unspecified service. Bytes
 
◆ Another unspecified service is located on port 8000, which may be http. It is not uncommon to find the http server on this port. Let's take a look:
 
$ Telnet 192.168.10.100 8000 Trying 192.168.1.100... connected to 192.168.1.100.Escape character is '^]'. GET/HTTP/1.0 HTTP/1.0 200 OKpragma: no-cacheContent-Type: text/htmlServer: MX4J-HTTPD/1.0 expires: nowCache-Control: no-cache...
 
This indicates that it is actually an HTTP server. In addition, we can use a Web browser to access this URL, or use the GET or HEAD command of Perl to verify it.
 
Apache Tomcat is running on port 8080.
 
Of course, this task can also be done through the security vulnerability scanner, provided that your scanner can identify the http [s] service running on a non-standard port. For example, Nessus can identify http [s] services on any port, test known Web server vulnerabilities, and test SSL configurations of https services. As previously suggested, Nessus can recognize popular application/Web interfaces, such as Tomcat management interfaces.
 
Solution 3
 
Multiple methods are available to identify the DNS names related to the specified IP address. The first method is the DNS region transfer method. The Domain Name Service saves a list of mappings between the host Name and IP address in the Domain. The "zone transfer" command allows the DNS server to return a list Of all domain names in the domain. Therefore, the DNS Zone Transfer can be used to discover hosts and routers in the domain. This technology is currently restricted because the DNS server does not accept partition transmission. However, it is worth a try. First, we must determine the DNS server that resolves the given IP address. If the symbolic name of the given address x. y. z. t is known, such as www.example.com, you can use tools such as nslookup, host, or dig to determine the name server. If you do not know the symbolic name of the given address, but the test target contains at least one symbolic name, you can try to query the name server, because the name server used by the given address also uses this name server. For example, if your audit object contains the IP address x. y. z. t and the name mail.example.com, you can find the name server used for the example.com domain.
 
The following example shows how to identify the name server for www.owasp.org. The host command used is as follows:
 
$ Host-t ns www.owasp.orgwww.owasp.org is an alias for owasp.org.owasp.org name server ns1.secure.net.owasp.org name server ns2.secure.net.
 
You can send a region transfer request to the name server that parses example.com. If you are lucky, you will get a column of DNS entries for this domain name. Including the obvious www.example.com and the less obvious webmail.example.com. Take a closer look at all the names returned by the region transfer and consider those that are related to the audit object.
 
You can try to request the region transfer for owasp.org: www.2cto.com
 
$ Host-l www.owasp.org ns1.secure. netUsing domain server: Name: ns1.secure. netAddress: 192.220.124.10 #53 Aliases: Host www.owasp.org not found: 5 (REFUSED); Transfer failed.
 
The second method is DNS reverse query. This process is very similar to the previous one, but it depends on reverse DNS records. Different from sending requests to a region, set the record type to PTR and query the given IP address. If you are lucky, a DNS name is returned. This technology relies entirely on the existence of IP-to-symbol name ing, but this ing does not always exist.
 
The third method is the Web-based DNS search method. This kind of search is similar to DNS region transfer, but the former relies entirely on Web-based services, an example of this kind of service is Netcraft's DNS search service, address http://searchdns.netcraft.com /? Host. You can query a column name that belongs to a selected domain name such as example.com. Then, you can check whether the obtained name is related to the audit object.
 
The fourth method is the reverse IP service. Reverse IP Service (view which domain names are shared under an IP address) or reverse IP Lookup, similar to reverse DNS lookup, the difference is that the query is a Web-based application rather than a name server. There are many such services available now. Because the returned results may be biased, it is best to use multiple services for comprehensive analysis. Here are some addresses:
 
Domain tools reverse IP: http://www.domaintools.com/reverse-ip/ (free registration required)
 
MSN search: http://search.msn.com Syntax: "ip: x. x" (no quotation marks)
 
Webhosting info: http://whois.webhosting.info/syntaxes: http://whois.webhosting.info/x.x.x.x
 
DNSstuff: http://www.dnsstuff.com/(multiple services available)
 
Http://net-square.com/msnpawn/index.shtml (installation required)
 
TomDNS: http://www.tomdns.net/(some services are still not public)
 
SEOlogs.com: http://www.seologs.com/ip-domains.html (reverse IP/Domain Name Lookup)
 
The fifth method is to use a search engine. After using the previous technology to collect information, you can use the search engine to assist in analysis. This further examines whether additional symbolic names are audit targets or applications that can be accessed through non-obvious URLs. For example, www.owasp.org is used as an example. You can search by Google or other search engines to find information related to the newly discovered domain names webgoat.org, webscarab.com, and webscarab.net. The technology for using search engines has been introduced earlier. Ii. Test the error code
 
When we perform penetration tests on Web applications, we usually encounter many error codes generated by applications or Web servers. To display the code, you need to use special requests or specially crafted tools. These codes are useful to penetration testers because they reveal a large amount of information about databases, defects, and other components that are directly linked to Web applications. This section analyzes how common error codes are used for vulnerability assessment. Pay special attention to these error codes when collecting information, because they are very helpful for the next step-improving work efficiency and reducing the total test time.
 
A common error during the search is HTTP 404 Not Found. Generally, this Code provides detailed information about the underlying Web server and related components. For example:
 
Not FoundThe requested URL/page.html was not found on this server. Apache/2.2.3 (Unix) mod_ssl/2.2.3 OpenSSL/0.9.7g DAV/2 PHP/5.1.2 Server at localhost Port 80
 
This error message may be generated when a nonexistent URL is requested. It provides useful information such as the Web server version, operating system, module, and other related products. This information is very important when we investigate the types and versions of operating systems and applications.
 
During security audit, the Web server will not only return useful error information, for example, consider the following error information example:
 
Microsoft ole db Provider for ODBC Drivers (0x80004005) [DBNETLIB] [ConnectionOpen (Connect ()]-SQL server does not exist or access denied
 
What's going on? Don't worry. Let's introduce it step by step. In this example, 80004005 is an IIS error code, indicating that a connection to the relevant database cannot be established. In many cases, this error message provides detailed information about the database type. It usually provides the operating system used at the underlying layer. With this information, penetration testers can also plan corresponding policies.
 
By controlling the variables passed to the database connection string, we can get more detailed error code.
 
Microsoft ole db Provider for ODBC Drivers error '000000' [Microsoft] [ODBC Access 97 ODBC driver Driver] General error Unable to open registry key 'driverid'
 
In this example, we can see a common error that exposes the type and version of the database system and the registry key value of the Windows operating system on which it depends.
 
Now, let's take a practical example to test a Web application that fails to connect to the database server and does not properly handle exceptions. This may cause database name resolution problems, handling unexpected variable values or other network problems.
 
We have a web interface for database management. It is used as a front-end GUI to send database queries, create tables, and modify database fields. During the POST request containing the logon certificate, the Penetration Tester receives the following error message. This message indicates that a MySQL database server exists:
 
Microsoft ole db Provider for ODBC Drivers (0x80004005) [MySQL] [ODBC 3.51 Driver] Unknown MySQL server host
 
If we find that the HTML code on the login page contains an implicit field of the database IP address, we can try to change this value in the URL with the database server address, in order to make the application mistakenly believe that the login is successful.
 
In another example, if you know the database server used by the Web application, you can use this information to test the SQL injection attack or persistent XSS attack on the server.
 
Handle errors in IIS and ASP.net
 
ASP. net is a framework used by Microsoft to develop Web applications. IIS is one of the commonly used Web servers. For application errors, we should try to collect more, but it is impossible to overwrite every exception.
 
IIS uses a set of custom error pages (usually under c: \ winnt \ help \ iishelp \ common) to display errors such as "404 page not found" to users. These default pages can be modified and custom error messages can be configured for the IIS server. When IIS receives a request for An aspx page, it will pass it to the. net Framework.
 
In the. net Framework, there are multiple methods to handle these errors. In ASP. net, you can handle these errors in three ways:
 
1. In the Web. config customErrors Section
 
2. In global. asax Application_Error Sub
 
3. aspx in Page_Error sub or related codebehind pages
 
Use web. config to handle errors
 
 
 
If mode = "On", the custom error function is enabled. If mode = RemoteOnly, the custom error is displayed to remote Web application users. Users who access the server locally will see the full stack trace, but the custom error cannot be displayed.
 
All errors will result in a redirection, that is, being redirected to the resource specified by defaultRedirect (for example, myerrorpagedefault. Status Code 404 is processed by myerrorpagefor404.aspx.
 
Handle errors in Global. asax
 
When an error occurs, Application_Error is called, so developers can write code for error handling/Page redirection here.
 
Private Sub Application_Error (ByVal sender As Object, ByVal e As System. EventArgs) Handles MyBase. ErrorEnd Sub
 
Handle error messages in Page_Error sub
 
This is similar to an application error:
 
Private Sub Page_Error (ByVal sender As Object, ByVal e As System. EventArgs) Handles MyBase. ErrorEnd Sub
 
Error hierarchy in ASP. net
 
First, process Page_Error sub, then Application_Error sub of global. asax, and finally the customErrors section in the web. config file.
 
It is very difficult to collect information for Web applications with server-side technology. However, the information found is very useful for correctly performing the next test, for example, you need to use SQL injection attacks or cross-site scripting attacks for testing, and reduce the false positive rate.
 
 
How to test ASP.net and IIS error handling
 
Start the browser and enter a random page Name:
 
Http: \ www.mywebserver.com \ anyrandomname. asp
 
If the server returns the following content:
 
The page cannot be foundHTTP 404-File not foundInternet Information Services
 
This indicates that IIS does not have the custom error function. Note that the extension. asp should be tested against. net custom errors: enter a random page name in the browser, as long as the extension is aspx:
 
Http: \ www.mywebserver.com \ anyrandomname. aspx
 
If the server returns the following content:
 
Server Error in '/'application. ------------------------------------------------------------------------------ The resource cannot be found. description: HTTP 404. the resource you are looking for (or one of its dependencies) cocould have been removed, had its name changed, or is temporarily unavailable. please review the following URL and make sure that it is spelled correctly.
 
This indicates that no error message is customized for. net configuration.
 
Next we will introduce the corresponding black box testing and examples:
 
Test method:
 
Telnet 80GET/HTTP/1.1
 
Returned results:
 
HTTP/1.1 404 Not FoundDate: Sat, 04 Nov 2006 15:26:48 GMTServer: Apache/2.2.3 (Unix) mod_ssl/2.2.3 OpenSSL/0.9.7gContent-Length: 310 Connection: closeContent-Type: text/html; charsets = iso-8859-1
 
Test method:
 
1. Network Problems
 
2. Improper configuration of host database addresses
 
Returned results:
 
Microsoft ole db Provider for ODBC Drivers (0x80004005) '[MySQL] [ODBC 3.51 Driver] Unknown MySQL server host
 
Iii. Summary
 
When conducting a security penetration test, we first need to collect as much information as possible for the target application. This article describes in detail how to test which applications are running on the target address and how to use messages in advance through error messages.