Runtime Protection Benchmarking

Overview

In the industry’s first test to closely emulate real world conditions, ShiftLeft subjected its runtime protection capabilities to a 14-day penetration test from Cobalt.io and is now publishing the results.

What makes this test unique?

In the real world, a customer deploys an application in the public cloud and hackers hack it for personal gain. Any device protecting the application cannot anticipate what the hackers will do and what techniques they will use.

Any lab-based testing, therefore, cannot emulate the unpredictable nature of the hackers. Moreover, tools used in such lab testing, e.g., Burp Suite, are also available to the vendors and are often part of their QA testing, which further reduces the variance presented by lab testing.

In this test, ShiftLeft selected Cobalt.io—a Pen Testing as a Service Platform—to conduct the test, emulating the real-world.

ShiftLeft Benchmark Penetration Test Report

Read the full penetration test report to compare the unprotected application & ShiftLeft protected application results

quote

ShiftLeft's ability to analyze an application in development, in order to automatically protect it in production, enables the company to benchmark themselves in unique ways that align well with the fast pace of the modern software development lifecycle.
quote

-Vik Phatak

NSS Labs | CEO

Testing methodology

The test deployed two identical applications. Both applications had the same vulnerabilities. One instance of the application was unprotected and the other was protected by ShiftLeft.

In order to accurately test ShiftLeft’s runtime protection capabilities, we created three teams that were isolated from each other: development, security, and penetration testing.

DevelopmentThe development team built a Java application with vulnerabilities in it representing the OWASP Top 10.

SecurityThe Security team relied solely on ShiftLeft’s ability to identify vulnerabilities in source code and protect them in runtime.

Penetration TestingCobalt.io performed the penetration testing.

Test application

The test application is a simple REST-based multi-tenant application emulating the functions of a retail-banking interface, including routes. The application was built with examples of six (6) of the relevant OWASP Top 10 vulnerabilities embedded into it. Hence, if the penetration testing team were able to breach runtime protection, the application would be exploitable. In order to make the application more vulnerable, it was built without support for any authorization or authentication schemes across tenants. All endpoints specified above can be exercised by any user. Hence, cross-site request forgery (CSRF) and the resulting cross-site scripting (XSS) were inherently out of scope.

Routes

  • GET /account
  • GET /account/:id
  • POST /account
  • POST /checkAccount
  • POST /checkAccountSimple
  • POST /account/:id/deposit
  • POST /account/:id/withdraw
  • POST /account/:id/addInterest
  • GET /rawcustomersbyname/:firstName
  • GET /customers/:id
  • PUT /customers/:id
  • DELETE /customers/:id
  • GET /customers
  • GET /createCustomer
  • GET /customersXML
  • POST /customers

Cobalt.io: Pen Testing as a Service

To ensure that the pen test was unbiased and thorough, ShiftLeft sought the expertise of a third-party pen testing solution. After evaluating the top approaches and vendors in the market, ShiftLeft choose Cobalt.io— a Pen Testing as a Service Platform—for its superior expertise, technology-driven platform, and efficient workflows. Additional benefits included:

  • Dynamic Reporting
  • Creative Results
  • Collaborative Team Approach

Cobalt.io tested both the protected and unprotected versions of the application over a 14-day period.

quote

After discovering several vulns in the unprotected application, our experts could no longer exploit in-scope vulns with the ShiftLeft protection in place.
quote

- Brian Levin

Services at Cobalt.io | VP

Results overview

Cobalt.io identified and exploited all the vulnerabilities in the unprotected application. When the application was retested with the protection offered by the ShiftLeft microagent, ShiftLeft blocked all attempted exploits across all vulnerabilities comprehensively protecting the application.

OWASP Category Vulnerability Type Endpoint Unprotected Application Protected Application
      Identified Exploited Identified Exploited
A1-Injection SQLi /rawcustomersbyname YES YES YES NO
A2-Broken Authentication Appropriate cookie protection /admin YES YES YES NO
A4-XML External Entities XXE /customersXML YES YES YES NO
A5-Broken Access Control Path traversal /saveSettings YES YES YES NO
A8-Insecure Deserialization Java deserialization /check YES YES YES NO
A9-Known vulnerabilities Known OSS vulnerability /checkFast YES YES YES NO

Real-World Runtime Protection Benchmarking

A Detailed analysis of how ShiftLeft's Protection Capabilities in Real-World Testing Scenarios