PTaaS - A Better Penetration Testing Delivery Model?
30 minutes for scope collection, free proposal, peace of mind. Fill out the form below.

Your Email or Phone:

Enter code:

Penetration Testing as a Service (PTaaS)

penetration testing price quote

Home - What - Why Pen Test - Why High Bit - Types - Reports - PTaaS - How Much?

Penetration Testing as a Service (PTaaS). A continual delivery model.

This page is not intended to enumerate the specifics for any type of penetration testing we perform. The specifics regarding types of penetration tests and the methodologies we employ are fully described elsewhere. This page describes an alternative delivery model, Penetration Testing as a Service, or PTaaS, and is applicable to all of the types of testing we perform, though we will use external web application testing for most of our examples.

What is PTaaS? First, to eliminate some possible confusion, this acronym is used in two ways relevant to IT testing. It can also mean 'Performance Testing as a Service', which is an entireley different thing and has nothing to do with penetration testing. The acronym is also used for 'Penetration Testing as a Service', which is the context treated here. This acronym, in a penetration testing context, has come into prevalent use in the last couple of years. In fact, penetration testing has always been a service, the difference is how that service is delivered. In this article, we'll explain the difference between PTaaS as a conceptual delivery model, and the traditional penetration testing delivery model.

When done responsibly, there is little difference in testing methodology and no difference in quality, but Penetration Testing as a Service (PTaaS) will typically employ a different billing, scheduling and delivery model than traditional penetration testing. These differences make PTaaS an attractive approach for some clients. It is attractive to many clients who perform or intend to perform penetration testing on a regular basis. PTaaS can be even more attractive to clients who produce B2B solutions and who are often asked by their own prospective clients, at unpredictable times, to provide evidence of penetration testing. Sometimes the fastest way to understand a concept is to examine the pros and cons, and that is the approach we will use here.

Here are some of the advantages of PTaaS:

  • Flatter spend. Most PTaaS offers are structured on a monthly billing cycle. This helps long term budgeting by flattening your spend and can make budget approval much easier, since it is a regular and expected expense.

  • Reduced administrative overhead for both the client and for the vendor. It takes time from both client and vendor decision makers just to confirm scope, pricing and terms for traditional model engagements. This factor can become more significant when you don't know the timing in advance. One such case occurs when an important prospect asks for evidence of penetration testing. In this situation, the need for penetration testing is at least partly driven by an immediate sales need, unpredictable in advance, and it puts timing pressure on decision makers. PTaaS is continual, ongoing and requires no further scope approvals unless you make changes to your scope.

  • Near continual monitoring and testing of networks and applications. The exact monitoring will depend on your scope, but typically includes an initial baseline engagement followed by monthly monitoring of all scoped networks, hosts and web applications. The monitoring includes examination for changes in exposed services as well as new vulnerabilities impacting existing services.

  • Much shorter, almost real time, remediation cycles. The focus of PTaaS is on rapid and direct communication, rather than detailed, polished reporting. In our case, when we discover a new vulnerability, whether reported by a scan tool or discovered manually, we upload a simple report and notify you. Then we get on the phone and explain it (if needed) rather than spend a lot of time creating detailed reports with screen captures and long narratives. This allows you to immediately know what was found, and immediately begin a conversation focused on correcting it and confirming the correction.

  • Early release detection and remediation. Depending on your compliance mandates, if any, and your architecture and SDLC model, it may be possible to deploy application changes to a security testing instance, which can be deployed wherever you want it in your SDLC release strategy, such as before any user acceptance or staging release. Detection and remediation can then occur prior to any further release. This approach can reduce the overall cost and exposure arising from unintended vulnerabilities.

  • Immediate availability of current, public facing reports. When your important new prospect asks your sales team when your last penetration test was conducted, your sales team can tell them ‘last month’ or ‘last week’, and hand them your current report. This is a big advantage for our clients who are in the B2B space. Prospect requests for evidence of penetration testing is the single biggest driver of first time penetration testing in this space. It is also the single biggest factor driving short engagement windows, usually paid for at a premium. The PTaaS model allows B2B clients to present that evidence very quickly, and demonstrates a commitment to continual penetration testing.

  • Same quality. When done responsibly, you should still receive full, quality, manual testing. The diffence should be in the billing, frequency, and reporting. In our case, we will use automation to monitor your environment for changes and identify vulnerabilities that can be identified with automation, but any detected changes also get manual attention. Ongoing testing activity is change driven, but you also have an annual full, no assumptions, manual review of your scoped networks, hosts and web applications.

  • Ongoing documentation. Your vendor should be providing log entries, and those log entries should be done by qualified testers who are actually monitoring and testing your environment. Every time our testers run or review a monthly change scan, the tester creates a log entry with the date, time, testers name and review status. If changes occur and no security flaws are found, or if new vulnerabilities are found on existing, unchanged systems, it is noted. If flaws are found and verified, a finding report is created, you are notified, and the current remediation status is noted and tracked. You won't have a final report like you would have with a traditional model, but unlike the traditional model, you do have a running record of what has been done, when it was done, who did it and the current status of any open findings, all the time. For some clients, this has more value than a snapshot final report in a traditional model.

There are also disadvantages with the PTaaS model. Nothing is perfect for everyone, and PTaaS is no exception. Here are some of the factors that make PTaaS less desirable than a traditional delivery model for some clients:

  • No final report. This is a plus for some, a minus for others. Under PTaaS, testing is ongoing and never completes, so there is a running record but no final report. For some clients, especially those who are meeting compliance mandates that require detailed explanations about testing approach and technical summaries targeted for auditors, or intended to cover a specific ‘point in time’ snapshot, this can be important and requires a more traditional approach to reporting. With PTaaS, there is no detailed, private, final report. There is an ongoing public facing report, which, like all reports intended for public consumption, does not include finding details. The details are in finding reports that are generated individually, as they are identified, are much less formal and are geared toward rapid communication and remediation. There is no final report that ties all of the findings together in context at any historic point in time. There is a running record of where you are right now, and plenty of history and logging, but the overall context never gets wrapped up and neatly tied together. This is perfect for some clients, but may be less than desirable for others.

  • Detrimental to a depth first approach. Like every other discipline, penetration testing has it's differing philosophies, and one of the more contentious and ongoing debates arises from depth versus breadth. This is not the place to dive into that debate, but PTaaS does have an impact here, especially if you lean toward a depth first philosophy. Covering the arguments surrounding this debate could easily consume hundreds of pages, but one factor is undeniable: penetrating deeply into an environment almost always involves chaining or combinining several vulnerabilities or even vectors together. This is something that is often done in traditional penetration testing, but this means the tester waited before reporting at least some of the findings. Testers don't get to combine flaws very often under the PTaaS model, unless the flaws occur together in the same release. Even then, the PTaaS model emphasizes immediate reporting and remediation, not waiting to report the finding because you want to try to combine it. Most clients who are interested in PTaaS have no position on this issue and when explained, lean toward a breadth first approach anyway, but if you are inclined toward depth first testing, PTaaS may not be for you.

  • Third party timing restrictions. Some hosting providers require permission before you can perform penetration testing of any kind. One major provider with such a policy is Amazon Web Services. AWS requires that you obtain authorization before testing, and the maximum window they allow, as of this writing, is 12 weeks. This policy appears to be based on a traditional delivery model for penetration testing, and assumes 12 weeks should be sufficient. This does not mean that you cannot use PTaaS in an AWS environment, but it does mean that you need to resubmit for approval 4 or 5 times per year. AWS may adapt their policy to allow for continual PTaaS delivery models, but as of this writing, 12 weeks is their maximum and reapprovals will be necessary. Other providers also have restrictions. You will need to check your contracts for restrictions no matter which approach to penetration testing you take, but short authorization windows can be a recurring annoyance under the PTaaS model.

  • Sensitive data handling and retention. Vendors will differ in their handling of sensitive data, but if they have solid practices for archiving and encrypting engagement data, with solid key management practices, PTaaS will cause complications. A PTaaS model requires that automated change detection and vulnerability scanning solutions run on regular basis. It is a rapid and continuing need, and is usually managed in a scheduling queue. Your data should still be encrypted and subject to access controls, but some of that data is in a more or less continually live state by necessity, and is seldom static long enough to be archived under key, especially with solid key management policies like split knowledge of keys, as you may be accustomed to with a traditional penetration testing engagement.

  • Altered applications. Some applications, especially those requiring multiple factor authentication or CAPTCHA protected login functions, may need to be modified in order to allow change detection and scanning software to run unattended. Basically, any control that is intended to prevent automation could, not surprisingly, cause problems with automated software. To balance this, methods that are often problematic for normal human users, such as source IP restrictions, can be used. Automated change detection and scanning traffic should always come from known IP addresses and can use source filters in addition to authentication mechanisms more suitable to automation. You may also benefit from updating version banners when you release, and placing those banners where they are visible prior to authentication, so that first order processes can move recently changed applications to the top of the queue for closer inspection. The essential point is, some application modifications may be necessary or beneficial, to allow automated systems to monitor for changes.

  • Scope creep. It has to be said, any agreement that sets a price based on a recurring service that never expires, has to deal with the issue of scope creep. If your applications are changing and new functionality is added, that is expected to a certain degree and is one of the advantages of the PTaaS model. If the application starts to resemble several applications bundled together, or starts using non-standard components that were not originally considered and priced, or adds web services that require special testing, or in general, becomes an application that would have been priced differently in an original quote, you should expect increased pricing. Most of our clients who are interested in the PTaaS model are very familiar with the concept of scope creep as it applies in software development, and understand that major changes may require price adjustment in penetration testing as well. Nonetheless, if it happens a lot it can eliminate some of the advantages of PTaaS by requiring recurring administrative overhead and budget approval. Some clients with rapidly evolving (not just growing) applications may be better served by a traditional model.

Things to be wary of with PTaaS offerings

Finally, and this is neither an advantage or a disadvantage, but a caution: PTaaS lends itself well to deception. There are two deceptions in particular that are made much easier under a PTaaS model:

  • Disguising a product as a service. There have always been pentetration testing vendors, some with big names, who have been willing to 'dress up' automated scan results and sell those results as a quality penetration test. If that meets your needs, no harm no foul, but if you are looking for quality manual penetration testing be wary. The Penetration Testing as a Service delivery model, with it's emphasis on easily consumable, near real time, abbreviated reports, makes this deception even easier than it is under the traditional approach. The screen captures, detailed narrative explanations and supporting documentation that are typical of quality manual testing under a traditional model are missing by design in the PTaaS delivery model.

    If you are not careful, you could be buying nothing more than automated scans created by a scan product, with the only service being delivery of automated reports over a web portal. Solutions like this deliver no actual manual penetration testing service at all, at most just a team of consultants who answer occasional questions about scan results or do a little false positive removal and ad a little verbiage before you see those results.

    This kind of deception can end up costing you, even if you are prepared to accept automated testing and it seems to be cheaper. If you are B2B and are testing to satisfy a request from a prospect or client, check to make sure that your client is willing to accept automated testing results, many are not. If you are testing for a compliance mandate, you should check the standard - PCI-DSS as of version 3, for instance, specifically requires manual testing.

  • Hiding underqualified personel. This is another deception, quite common in traditional penetration testing, made even easier under PTaaS. The PTaaS model seems to justify putting a web portal between you and the testing team. It's important to note that there really is nothing in the PTaaS delivery concept that even requires a portal, but the concept does lend well to a portal delivery. That portal can easily be utilized as a barrier to visibility. If the portal is your only communication channel, it is particularly easy to hide the identity, and qualifications, of the testing team members. You could end up with a team of consultants who may or may not have any penetration testing certifications, who could be anywhere in the world, identified only by a name and a photo, who only communicate with you over a portal.

Before contract signing, be especially wary of:

  • Sales documentation using terms like researchers, and consultants, rather than certified penetration testers. PTaaS is a great model for hiding underqualified resources.

  • Sales documentation emphasizing your access to qualified personel. Access to the tester should never be in question because it is the tester who should be contacting you. Make sure the access includes the testers company email address and a phone number, not just access through your portal account.

  • Contracts that do not clearly specify actual, manual penetration testing work by personel with the right qualifications, or contracts that seem to imply it without actually specifying any manual work.

  • Long term contracts without an easy cancellation.

After contract signing, assuming you have an easy cancellation clause (which you should have), be wary if:

  • In practice, the service does not seem to provide log entries generated by humans, and at least occasional screen captures and supporting files, along with manually generated notes, at the time of the initial reporting of a finding.

  • Your follow up questions for specific findings seem to be answered in general terms, by someone who seems to lack specific knowledge of your environment or applications.

  • You get a different tester every time. This may mean that the vendor is crowd sourcing your testing. Some clients are comfortable sharing their most sensitive data with a revolving door of global contactors, but most clients are not, and it should at least have been disclosed in advance.

  • You are constantly wading through false positives, or always seem to be the one who has to contact the vendor for clarification, instead of the other way around.

  • You are only able to contact the tester through service portal interaction, and are not provided with the tester's company email address, phone number or other contact channels.

If you find yourself in an uncomfortable relationship, that's what the cancellation clause is for.

Penetration Testing as a Service (PTaaS) Summary.

The PTaaS delivery model is focused on nearly continual change monitoring, testing and rapid reporting and communication. It sacrifices polished, detailed reporting in favor of rapid remediation and ongoing testing. It reduces administrative overhead, can shorten remediation cycles and catch flaws earlier in the release cycle. It can have strong sales advantages for B2B clients, flattens budgeting and can simplify budget approval. It is not cheaper when done correctly, because the same manual effort is required from the same highly qualified testers, but it may increase ROI for the same spend, depending on your needs. For many of our clients, this is a delivery model better suited to their needs than a traditional model.

The approach can also cause problems if you need detailed reporting, can impact sensitive data handling and retention, may be complicated by third party approval factors, may require changes to applications and is not compatible with all penetration testing philosophies. For some clients, the traditional model is still the best.

If you do decide that PTaaS is the right approach for you, you still need to be careful that you are actually getting what you think you are getting, and should insist on a contract that allows you to bail out easily. In our case, for example, our PTaaS contracts allow you to cancel at any time, and we insist that we start out with a traditional engagement so that you get to know us and our penetration testers, and we get to know you and your systems. For us, this is a relationship and not just a service.

If you are interested in the PTaaS delivery model, please ask us about it. This has been brief treatment of the subject and we will be happy to further explore the options with you.

Ask us for a free, quick, no hassle quote using the contact form above.