It’s not uncommon for teams to commission a pen test (particularly their first one) without really knowing what’s about to happen.

The process can feel like a black box, which is ironic given that black box testing isn’t where you should be heading. This lack of clarity and understanding is not surprising given the technical nature of these engagements. However, it doesn’t need to be like this.

In this guide, we walk you through what actually happens, from initial conversation to ongoing assurance.

Our view is that an informed buyer makes better decisions and gets more from the testing experience and outcomes.

Before the Test Begins: Scoping

A pen test doesn’t start when the tester opens their laptop and accesses the testing targets, it starts much earlier with the scoping conversations. What gets agreed at this stage includes the targets to be tested, the testing standard(s) or references, the customer’s goals and expectations for the testing, the nature of the testing environments, timings for the test, and other key details such policies and practices with customer source code.

Planning a test against a testing environment that mirrors production as closely as possible is important. Testing on a production system is to be avoided due to the offensive nature of pen testing. Typically a testing or staging environment that most closely mirrors your production environment is best because it enables findings to be directly applicable to production, but without risking production during the testing. This similarity includes not only the codebase but also the infrastructure on which the application is deployed. Any differences between the production and the test target systems can lead to false-positive and/or false-negative findings. Slow test target systems can also affect testing time, so it’s important for best use of time and budget that the test target system has comparable performance to the production system(s).

Source code access is also usually agreed. This enables source code supported (and therefore standards-based) testing and analysis to be undertaken. This is a more efficient and effective way to undertake testing, allowing testers to look for and find patterns across the codebase and to provide more targeted proofs of concept (PoCs). This approach provides better value, since it allows testers to find and report more items within the allocated testing time and budget (such as all examples of a category of vulnerability across the entire application).

The testing methodology also gets agreed at this stage (either standards-based, for example using the OWASP ASVS, or more general testing, using the OWASP Top 10). It is key that testing methodology is right-sized for your needs and the target that’s being tested.

The Test Itself

Proper pen testing is manual by nature. There are of course automated tools that are used, and increasingly these have AI capabilities. These tools are evolving rapidly, and allow testers to scan source code more quickly, attempt more varied attack scenarios, and add speed around chaining of findings. But testing is still managed by experienced humans, and individual judgment and oversight are still at the core of a proper manual pen test.

During testing, testers are looking carefully for vulnerabilities across the target systems. Depending on the test being undertaken these might be issues such as broken access control, authentication, data leakage across tenants, and SQL injection, for example.

Testing standards can require quite a bit of Q&A with your development team during the course of the test, this is normal and beneficial, not a sign something’s wrong.

As the testing progresses, evidence of vulnerabilities and PoC details are collected. These are included in the report to allow you to be able to reproduce, verify, and fully understand every vulnerability.

In terms of the customer experience during the testing period, aside from some Q&A with your developers, often it can seem unsettlingly quiet. But this just means that the testers are working unhindered and are not facing any issues or blocks.

Any critical issues are reported as soon as they are discovered and adequately characterised, usually same day, but all other findings are generally not reported until the testing window is complete because this enables the vulnerability to be fully profiled both individually and with reference to all other discovered vulnerabilities. For example, there might be a chain of attack which moves from one discovered vulnerability to another, so it’s important to wait until testing is complete before sharing details of all vulnerabilities with you.

A test can take anything from a few days to many weeks. This ultimately depends on the quantity, size and complexity of the targets, and the methodology that is being used for testing. Standards-based testing is undertaken in a methodical, audit style fashion so is more time consuming, but it does provide a higher level of assurance.

What You Receive: The Initial Report

Once testing is complete the findings are shared securely. A good report will include details on every vulnerability found, along with its severity rating, risk rating, and PoC details that allow reproduction and verification of each finding.

Any severity rating should be set using an accepted industry system. This is important because inflating findings to look thorough (or downplaying them to look good) does not serve you, your internal teams, or your stakeholders.

A quality report will include not just the findings but expert judgment on what those findings mean for your business and the risks they pose. The report must be written in a clear and accessible style so that your development team can act on it, and your board can understand the headlines.

Remediate and Re-Test

Your report lands. Now what?

Ideally your team triages and then remediates the issues as soon as reasonably practicable, with assistance from the testers if needed. Once fixes have been applied, the testers jump in again at an agreed time, undertaking targeted re-testing of the issues. This re-testing ideally allows the testers to confirm that everything has been resolved or addressed according to the recommendations in the report.

The final step in a round of testing is the issuance of a further low details document that describes what was tested, when and how, and by whom (and holding which certifications in relation to the applicable testing standards). This document should not contain any sensitive information about the findings or targets, ensuring that it can be safely shared with non-technical and external stakeholders like governance boards, auditors, existing and potential customers, and during compliance conversations.

This final output should empower you to move from “trust us” to “here’s the evidence”. This is the deliverable that changes conversations.

Beyond the Testing Engagement

A great pen testing engagement doesn’t end at delivery of the report, it should potentially also change how your team builds, configures and deploys going forward. Key learnings are fed into DevSecOps practices and test suites to avoid re-introducing these same categories of issues in the future. This makes the difference between a one-off box-ticking engagement and building genuine improvements in your Secure Software Development Lifecycle (SSDLC) and your organisation’s security posture over time.

Continuous assurance from your pen testing provider allows for ad hoc testing to be undertaken between scheduled ongoing tests, for example prior to a significant release or additional functionality being added or refactored. This is the compounding value of periodic testing where each step builds on the last, on a journey of security maturity.

Ultimately, a great pen test should leave you more informed, not just more compliant (or worst case with a false sense of confidence). The goal here is confidence you can defend, grounded in globally accepted standards, and with clear evidence to back that up.