Runtime Protection Benchmarking


In the industry’s first test to closely emulate real world conditions, ShiftLeft subjected its runtime protection capabilities to a 14-day penetration test from and is now publishing the results.

What makes this test unique?

In the real world, a customer deploys an application in the public cloud and hackers hack it for personal gain. Any device protecting the application cannot anticipate what the hackers will do and what techniques they will use.

Any lab-based testing, therefore, cannot emulate the unpredictable nature of the hackers. Moreover, tools used in such lab testing, e.g., Burp Suite, are also available to the vendors and are often part of their QA testing, which further reduces the variance presented by lab testing.

In this test, ShiftLeft selected—a Pen Testing as a Service Platform—to conduct the test, emulating the real-world.

ShiftLeft Benchmark Penetration Test Report

Read the full penetration test report to compare the unprotected application & ShiftLeft protected application results


ShiftLeft's ability to analyze an application in development, in order to automatically protect it in production, enables the company to benchmark themselves in unique ways that align well with the fast pace of the modern software development lifecycle.

-Vik Phatak

NSS Labs | CEO

Testing methodology

The test deployed two identical applications. Both applications had the same vulnerabilities. One instance of the application was unprotected and the other was protected by ShiftLeft.

In order to accurately test ShiftLeft’s runtime protection capabilities, we created three teams that were isolated from each other: development, security, and penetration testing.

DevelopmentThe development team built a Java application with vulnerabilities in it representing the OWASP Top 10.

SecurityThe Security team relied solely on ShiftLeft’s ability to identify vulnerabilities in source code and protect them in runtime.

Penetration performed the penetration testing.

Test application

The test application is a simple REST-based multi-tenant application emulating the functions of a retail-banking interface, including routes. The application was built with examples of six (6) of the relevant OWASP Top 10 vulnerabilities embedded into it. Hence, if the penetration testing team were able to breach runtime protection, the application would be exploitable. In order to make the application more vulnerable, it was built without support for any authorization or authentication schemes across tenants. All endpoints specified above can be exercised by any user. Hence, cross-site request forgery (CSRF) and the resulting cross-site scripting (XSS) were inherently out of scope.


  • GET /account
  • GET /account/:id
  • POST /account
  • POST /checkAccount
  • POST /checkAccountSimple
  • POST /account/:id/deposit
  • POST /account/:id/withdraw
  • POST /account/:id/addInterest
  • GET /rawcustomersbyname/:firstName
  • GET /customers/:id
  • PUT /customers/:id
  • DELETE /customers/:id
  • GET /customers
  • GET /createCustomer
  • GET /customersXML
  • POST /customers Pen Testing as a Service

To ensure that the pen test was unbiased and thorough, ShiftLeft sought the expertise of a third-party pen testing solution. After evaluating the top approaches and vendors in the market, ShiftLeft choose— a Pen Testing as a Service Platform—for its superior expertise, technology-driven platform, and efficient workflows. Additional benefits included:

  • Dynamic Reporting
  • Creative Results
  • Collaborative Team Approach tested both the protected and unprotected versions of the application over a 14-day period.


After discovering several vulns in the unprotected application, our experts could no longer exploit in-scope vulns with the ShiftLeft protection in place.

- Brian Levin

Services at | VP

Results overview identified and exploited all the vulnerabilities in the unprotected application. When the application was retested with the protection of ShiftLeft, only 1 vulnerability was visible to the pen testers. When ShiftLeft's microagent was added, this blocked all attempted exploits of the lone vulnerability.

OWASP Category Vulnerability Type Endpoint Unprotected Application Protected Application
      Identified Exploited Identified Exploited
A1-Injection SQLi /rawcustomersbyname YES YES YES NO
A2-Broken Authentication Appropriate cookie protection /admin YES YES YES NO
A4-XML External Entities XXE /customersXML YES YES YES NO
A5-Broken Access Control Path traversal /saveSettings YES YES YES NO
A8-Insecure Deserialization Java deserialization /check YES YES YES NO
A9-Known vulnerabilities Known OSS vulnerability /checkFast YES YES YES NO

Real-World Runtime Protection Benchmarking

A Detailed analysis of how ShiftLeft's Protection Capabilities in Real-World Testing Scenarios