Pros and cons of manual vs. automated penetration testing

Manual pen testing pros and cons

The top benefits of manual pen testing are it offers flexibility and a higher likelihood of discovering and mitigating vulnerabilities within the tested systems. Manual pen testing can find cleverer vulnerabilities and attacks that automated tests may miss, such as blind SQL injection attacks, logic flaws and access control vulnerabilities. A trained professional can examine the responses of an application to such an attack in a manual pen test, potentially catching responses that may appear legitimate to automated software but, in reality, are a problem.

Some pen tests can also only be performed manually. If a company wants to examine social engineering preparedness, for example, manual pen testing is needed, especially when testing for vishing.

Manual pen testing can also enable more creativity when looking for flaws. "A good penetration tester will use their instincts and, based on the results, may opt to go into testing further in an unexpected direction," said Jon Oltsik, analyst at Enterprise Strategy Group, a division of TechTarget.

Another benefit of manual pen testing is having an expert on hand to review reports. While automated pen testing tools also generate reports, security analysts still have to review and remediate many of the issues detected.

The top cons of manual pen testing are cost and time. Depending on a pen test's thoroughness, it could take weeks to get results, which isn't always ideal -- especially if major vulnerabilities exist.

Manual pen testing can also be expensive, which is why many companies do it only to fulfill compliance and regulatory requirements. When companies can't afford an internal red team or pen testing team, third-party service providers are used for testing needs -- another cost.

Automated pen testing pros and cons

Pen testing is complicated and expensive, so many companies conduct tests infrequently. The benefits of less expensive and easier access to testing via automation could change that.

"There is an appetite from organizations to do more frequent testing," said Mitchell Schneider, analyst at Gartner. "One of the benefits we have seen from automated pen testing tools is an increase in testing frequency. Companies want to address pertinent risks and threats in a timely manner versus having to wait for a test to be scheduled."

Frequent automated pen testing also helps companies evaluate their entire computer systems, which may get updated -- for example, during rapid release cycles -- more often than testing occurs. "You need something that's automated to really get a view of the environment," said Jeff Pollard, analyst at Forrester Research.

Another benefit of automated pen testing is it frees up security analysts' time so they can focus their attention on other tasks that may get put on hold during testing periods. Automation can also handle repetitious tasks that aren't necessarily complicated but are time-consuming for humans to complete.

One potential con of automated pen testing is analysts still see it as an emerging market. "Standalone automated tools have evolved over the last few years," Oltsik said. "It's an innovative and growing market as venture capitalist investment continues."

Another downside of automation is testing results depend on how good the penetration tool itself is, as well as how knowledgeable the person using it is. "The baggage of automated testing is people," Oltsik said. "The software is only as good as your knowledge base. You've got to program in certain tactics and techniques for vulnerabilities." If the pen testing software developer didn't do their job well, for example, then the automated pen test is flawed and could miss critical issues.

Some also worry automated tools could displace human pen testers, but Oltsik said that's not necessarily the case. "It's possible that these tests get so good you'll just need overseers and auditors to managed automated tests," he said. "But I don't see that anytime in the near future."

Additionally, automated pen testing remains limited in function and cannot be deployed for every testing scenario. Pen tests on wireless networks, web apps and social engineering, for example, aren't supported by most tools.

Combining manual and automated pen testing

When it comes to choosing manual vs. automated pen testing, it's often not a question of either/or. Rather, automated pen testing tools should augment manual pen testing efforts.

Automated pen testing tools won't fully work for every type of pen test out there, Schneider said. "And, at least for the next few years, they will never fully replace a pen tester or red team," he added.

Another option automation has also enabled is penetration testing as a service (PTaaS). Some services are already available from vendors such as NetSPI, Cobalt and Pentest People. PTaaS offerings are a mix of manual and automated pen testing that make it easier for companies to fulfill specific pen testing needs, such as to satisfy compliance or regulatory requirements.