Penetration Testing Methodology

Home - Safety and Stability - Network - Web Applications - Wireless - Compliance -

Penetration Testing Safety and Stability – Reducing risk arising from the penetration test itself.

Our testing methodology is based on the widely accepted NIST SP800-115 'Technical Guide to Information Security Testing and Assessment ' at 5.2 'Penetration Testing', with certain modifications intended to reduce risk to production systems as suggested in NIST SP800-115 at 6.3 'Selecting and Customizing Techniques'.

The NIST reference to the 'risk to production systems' is no accident, and the use of 'customized techniques' as suggested by NIST is important. NIST recognizes that penetration testing, by it's nature, involves doing unexpected things to systems and applications. If it were known in advance how those systems and applications would behave under unusual circumstances, there would be little reason to perform a test at all. Unexpected input can be detrimental to your systems and business processes. NIST specifically recognizes this fact and suggests customizing penetration testing techniques to reduce that risk.

How does High Bit Security reduce the risk?

We build risk reduction into our methodology, enforced by internal policies and procedures, and allow departure from those procedures only in unusual circumstances and only when specifically authorized by our client.

After years of research into the issue of penetration testing risk, we believe that client risk exposure can be divided into three broad categories: System, Human, and Third Party. At the end of this section we describe specific actions we take to minimize these risks. First, here are some examples of things that go wrong with penetration testing, by category:

System Risks.

System risks include data corruption risks and stability risks, and can impact systems directly and indirectly. Direct impact occurs when a targeted system behaves in an unexpected and negative way. Here are a few examples of Direct Impact to systems:

Databases damaged when data records or referential integrity is destroyed by application scanners.
Routers, switches, and other critical systems crashed by network vulnerability scanners, or even simple port scans.
Systems that are simply overwhelmed by too much testing volume.

Direct, negative impact to systems can occur even with very well known and trusted tools used by almost every security firm, such as the venerable Nmap port scanner.

Indirect impact occurs when non-targeted systems or processes are impacted by testing activity directed at targeted systems. These indirect impact scenarios are likely to be overlooked in penetration testing planning. Here are a few examples of Indirect Impact to systems:

Shared resources. A common example is two websites that share the same back end database. Targeting of website A (in scope) results in negative impact to website B (out of scope) when a vulnerability in A leads to modifications in the shared database.
Batch jobs. In any case where a targeted system leaves a persistent record that is used by another process, there is a risk of unintended behavior in non-targeted systems due to unexpected input to those systems. Batch jobs or scheduled tasks are a classic case of unexpected penetration testing inputs potentially reaching non-targeted systems.
Uncontrollable Systems. One example of a system that is completely beyond your control is anti-spam software and black list databases managed and deployed by others. Application scanners can invoke email functionality in web applications that can result in hundreds or thousands of emails launched at broadly separated users of the application, which, in turn, can trip multiple instances of anti-spam filters owned by others, and result in public blacklisting of your email servers. Another example is search engine indexing. Websites can sometimes be defaced by application scanners and then immediately indexed by search engines. Even if the site is quickly cleaned up, the resulting search engine indexing can persist for a long time. A similar scenario can occur when scanners put a production web application into a persistent error state. Error messages with full stack traces may then be indexed by search engines, giving anyone on the planet a searchable and free look deep into the application structure, which is not what anyone hopes to achieve with a penetration test.

In today's interconnected world there are many ways that unexpected behavior from one system can propagate to others. In some cases those other systems are beyond your control and the effects can be difficult to reverse.

Human Risks.

The Human risk factors include productivity, morale and reputation risks. The most obvious impact on your human resources comes from testing that is intentionally directed at people. Social Engineering is the industry term for that kind of testing activity. It is not safe to assume however, that there are no human risks in the absence of Social Engineering. Here is an example of human impact caused by conventional testing activity:

One of the common causes of productivity loss occurs when websites that support email functionality are subjected to automated scanning activity. Organizational sales and support staffs can become virtually incapacitated when thousands of support tickets or sales inquiry emails begin to flood their in-boxes. The same can happen with text messaging, chat functions, or other side channel communication vectors.

Third Party Risks.

The last category, Third Party risks, includes both suitability and liability risks. Suitability failures occur when testing engagements fail to cover necessary scope or fail to meet other requirements imposed by third parties. When testing outcomes result in re-work, added cost or missed deadlines, it is appropriate to consider them as risks. The second risk factor in this category is Liability. When testing activity impacts third parties who were not a party to the governing engagement contract, it often brings significant liability exposure to all who were a party to the engagement contract. Here are several examples:

Social Engineering, malicious or self-propagating payloads. Imagine an infectious media test where a non-target person picks up a USB stick with a malicious payload, or imagine a malicious email payload received by a target and then forwarded to others. When this happens, people who are not part of your organization could be affected and bring adverse action against you, your vendor, or both. Dangerous social engineering payloads are found in standard social engineering tool kits used by many vendors. Even benign payloads can expose you to liability if those payloads are self-propagating. Your organization may need to invest considerable post-engagement time and effort to contain and eliminate self-propagating payloads from your environment. Now imagine that the organization absorbing this expense had no part in your engagement, and is cleaning up after payloads intended for your engagement have escaped from scope and self-propagated through their environment. The liability risk in poorly managed Social Engineering engagements can be substantial, especially where electronic vectors and payloads are used.
Other unapproved third party impact. Other penetration testing tools can impact targets that are not in your approved scope. This is a very common and preventable occurrence with web application scanners. Consider the following URL: http://www.yourtarget.com. When this URL is in your approved scope, it typically means that the URL and all resources under it are included in scope. This means that a resource at http://www.yourtarget.com/not-ours/ is in scope. If that resource includes functionality that impacts third parties, and your vendor fails to properly configure the scanner to exclude it from testing, unauthorized third party impact can occur. Other scope vagueness can result in problems as well, such as unspecified protocols. Does https://yourtarget.com expose the same functions and data as http://yourtarget.com, apart from the encryption? If your vendor encounters either case while testing the other, will they consider it to be in scope, or not? And what about web applications discovered to be running on servers that are in the approved scope by IP address, even though no web applications served by that host are specifically listed as a web application target? Knowing exactly how these situations will be handled is important in any penetration test, but any potential for third party impact increases your risk. We are very careful about understanding your scope.
Approved attacks on third parties. Unfortunately, it's not a misprint. Far too frequently, competent and careful penetration testers interact with systems that are not owned by the client, but were authorized by the client. The larger the scope and the larger the organization, the more likely it is for the scope list to include a resource that is no longer owned by the organization, or is leased or licensed to someone else, or leased or licensed by someone else to you, without approval for penetration testing.

Managing Risk.

Here are some of the things we do, by enforced policy and procedure, to reduce your risk:

1. Pre-Engagement Interviews. Minimizing direct and indirect system impact, human and third party risk cannot be accomplished through simple target definition in scope documents. Prevention requires careful consideration, an understanding of the risks and knowledge of potential problem areas. We will ask you for a pre-engagement conference with the personnel who are most likely to know about any shared resources or batch jobs, databases, and third party owned systems or relationships that might be adversely effected by penetration testing activity.

2. Automated Scanner Configuration. Automated scanners are indispensable to full test coverage yet scanners are one of the biggest risks to production systems. Our network vulnerability scanners use customized configurations designed to avoid denial of service tests and unsafe memory corruption testing, and are also configured to use low thread counts to avoid overwhelming target systems or network devices. Application scanners are similarly configured and tightly controlled for adherence to scope boundaries. These constraints add time to scanning tasks, and that is one of the reasons we ask for a longer engagement window than some other vendors. It takes time to do it carefully and avoid risk.

3. Canary Scans. We invented the term 'Canary Scan'. It is a technique that can save important production systems from exposure to potentially dangerous scanner activity. It is not always possible, but if you have non-production systems that are configured similarly to production systems, we can launch automated tools first at these less important targets, and then, if there are no negative consequences, proceed to use the tools on the actual targets of the engagement. Most of the time, there is no impact at all, but if there is a negative impact this technique can save important production systems from risk exposure. We will ask you about potential 'Canary Hosts' in our pre-engagement Interview.

4. Tester Awareness and Avoidance. Our testers perform a full manual review of all web applications before launching any automated tools and are trained to recognize potentially dangerous database situations, side channel vectors like email, chat and text messaging functions, and third party risk. They are trained to stop and request further authorization if they suspect that our activity might impact any of these areas. This policy does not completely eliminate risk, and it sometimes means unexpected delays, but has saved many production databases and prevented other unintended consequences.

5. Electronic Social Engineering. All electronic social engineering payloads used by High Bit Security are custom developed for your engagement, and are benign payloads (they cannot harm a system or even access a system) and are incapable of self-propagation. This approach virtually eliminates third party risk from electronic social engineering engagements.

6. Depth and Breadth Balance. This is an important factor in reducing the risk of penetration testing engagements, and represents a significant difference between High Bit Security and other vendors. Most vendors opt for one of two approaches to penetration testing. The Breadth First approach emphasizes thorough coverage, and practitioners tend to rely heavily on automated tools to accomplish all or much of the testing. This approach means a heavy reliance on the most dangerous tools – automated scanners. While generally cheaper this approach is dangerous and can miss important findings depending on the amount of manual testing employed. The Depth First approach emphasizes undetected penetration in depth, and practitioners of this approach tend to avoid the use of automated tools because those tools can draw attention. Depth First practitioners use far more manual effort and often consider an engagement to be unsuccessful if they are not able to fully compromise a system or expose deeper systems. This approach often results in inadequate breadth of coverage, is usually more expensive and can expose deeper systems to unnecessary risk. At High Bit Security, we employ an approach that balances depth and breadth. We use carefully configured automated tools to aid in breadth of testing, substituting manual testing methods in cases where automation is unsafe. We also use extensive manual effort to dig deep into potential security faults, but we stop pursuing depth when we have proven the fault, and have documented the finding in detail. This balanced approach allows for thorough breadth of coverage, sufficient depth, detailed documentation, and above all, safer testing because it reduces reliance on automated tools and does not expose deeper systems to needless risk. An example of a finding report documenting a blind SQL injection fault, and demonstrating this principle can be found here.

Safety and Stability Summary.

There is no such thing as a risk free penetration test. Penetration testing, by it's nature, involves doing unexpected things to systems and people. The risk can never be completely eliminated, but it can be reduced and controlled. High Bit Security has developed extensive policies and procedures after years of attention and commitment to the issue of risk, and actively enforces and uses these procedures on every engagement. We hope that the details given here have served to demonstrate our knowledge and commitment, and we look forward to working with you toward a thorough and safer engagement.

Ask us for a free, quick, no hassle quote using the contact form above.