Application Security test: Two-sided black box

The greatest risk to software security is the nature of the opacity of the tools and processes, and the potential for false negative errors to be covered by different inspection techniques (such as automated dynamic testing).

While the Security software Development Lifecycle (SDLC) has many relevant best practices, most organizations still have a tendency to rely primarily on testing to build secure software. The most serious side effect of the current test approach is that the organization is not sure what has been tested by its solution, and (even more importantly) what has not been tested. Our research shows that any single automated assurance mechanism can test up to 44% of security requirements. The NIST Static Analysis tool exposition found that there were 26 known vulnerabilities in Tomcat, but all static analysis tools were combined to report only 4 warnings. Because reliance on opaque testing is a pervasive habit that has even become an industry standard, many organizations are satisfied with testing as the primary means of building secure software.

For example, let's say you hire a consulting firm to perform penetration testing for your software. Many people refer to this test as a "black box" (based on a warranty technique with the same name), and testers do not have detailed system internals (such as system code). After the test is executed, a report is invariably generated that outlines several types of vulnerabilities in your application. You fix the vulnerability and then submit the application for regression testing, and the next report says "cleared"-that is, there are no holes. Or at best, just tell you that your application will not be compromised in the same way by the same testers at the same time. But on the other hand, it won't tell you:

What are some of the potential threats to your application?

What are some of the threats in your application that are "not easy to attack"?

What threats in your application have not been evaluated by the tester? From a run-time perspective, what threats are not tested?

How does the time of the test and other constraints affect the reliability of the results? For example, if testers have 5 days, what other security tests will they perform?

How high is the tester's skill level? Can you get a similar set of results from a different tester or another consulting firm?

In our experience, organizations are unable to answer most of these questions. Black boxes are two sides: on the one hand, testers do not know the internal structure of the application, on the other hand, the organization applying for testing is not aware of the security of its own software. It's not just that we're aware of the problem: Haroon Meer discussed the challenge of penetration testing on 44con. Most of these issues apply to any form of validation: Automated dynamic testing, automated static testing, manual penetration testing, and manual code review. In fact, a recent paper has introduced similar challenges in source code review.

An example of a requirement

To better illustrate this issue, let's look at some common security requirements for high-risk software and how to apply common validation methods to these requirements.

Requirements: hash (hash) user password using a secure hash algorithm (such as SHA-2) and a unique obfuscation value (salt value). Multiple iterations of the algorithm.

In the past year, there have been well-known password leaks in LinkedIn, Last FM and Twitter, and this requirement is specific and timely for such flaws.

How to apply common authentication methods:

Automated run-time test: It is not possible to access the saved password, so this method cannot be used to validate this requirement

Manual Runtime Test: This method can be used to validate this requirement only if other development results in a stored password dump. This is not something you can control, so you can't rely on run-time testing to validate this requirement.

Automated Static Analysis: This method can be used to validate this requirement only if the following conditions are met:

Tools clear how identity authentication works (for example, using standard components, like the Java Realms)

The tool knows which particular hashing algorithm the application uses

If the application uses a unique obfuscation value for each hash, the tool must clearly confuse the algorithm and confuse the value