ExpressVPN leak testing tools

The ExpressVPN Leak Testing Tools are an extensible suite of Python tools designed for both manual and automated leak testing of VPN applications.

Developed by ExpressVPN security researchers, the tools were initially built for two purposes:

• Providing robust, automated regression testing of ExpressVPN applications
• Facilitating leak testing investigations to strengthen those applications further

We soon came to realize that these tools could actually be beneficial to the broader VPN industry. We thus decided to make them fully available open source under the MIT License.

We hope that by doing so that we will help provide a more private and secure online experience for all VPN users by:

• giving users and third parties an independent way to verify the quality of VPNs and assess their own privacy and security risks;
• creating a higher benchmark for protection against leaks for the entire VPN industry; and
• bringing together expertise from the whole industry around leak testing

Getting started with the tools

The tools can be found on GitHub at: https://github.com/expressvpn/expressvpn_leak_testing.

To get started, follow the instructions in the README.md file. While we provide an overview of the tools below, the GitHub repository should always be considered the single source of truth for documentation relating to the tools.

Note that to follow the testing instructions included in our Privacy Research Lab case studies, you will need to install the leak tools as per the documentation in GitHub. It is assumed that whenever you run any of the case studies, the tools root directory refers to the location where you have cloned the above git repository.

Overview of the tools

We use Python

The tools are written primarily in Python, with a few helper tools in other languages as necessary. We chose Python for several reasons:

• It’s cross-platform and supported out-of-the-box in most cases
• It’s well known to a large percentage of developers and is thus readily accessible
• It’s fast to develop in—we have no performance requirements for a testing framework
• It’s an expressive language which allows us to write easily readable code

Note that we use Python 3, as this is fast becoming the de facto standard.

It’s like unit testing

The tools were designed to run like a unit testing suite. A single script is used to launch the suite which then runs a series of tests. Each test checks for a specific type of leak under specific conditions, e.g., under certain network conditions, then reports either a pass, fail, or error.

A test—or test case—roughly corresponds to a single Python file. The test suite was designed to be familiar to developers used to working with unit testing suites, so each test follows the familiar pattern of setup(), test(), and teardown().

The test cases are designed to be as generic as possible. Test cases can be run with different VPN configurations, different network configurations, and so forth. When they are run, they are provided with a configuration which specifies how they will run for that particular round of testing. A configuration file is passed to the test suite which lists all the tests to run and their configurations.

An example of how to run the tests would be to execute the following command from the tools root directory:

./run_tests.sh -c configs/case_studies/vanilla_leaks.py

Device and Network Configurations

The suite has been designed to be able to run tests on arbitrary configurations of devices and networks. We were aware from the very beginning that some aspects of VPN behavior would need more than one device to test, so we designed the test suite to enable this from the outset.

The device from which the tests are run is referred to as the test orchestrator. All other devices are simply referred to as test devices.

The simplest type of test runs on a single device (single device tests) and test for leaks directly on that device. For example, one can run a wide class of Windows leak tests directly on a Windows device. In this case, the test orchestrator is actually also a test device. More complex tests (multi-device tests) might involve a test device running the VPN, a router, and a packet capture device.

Automation

The test suite was designed to fully support the automation of tests. However, out of the box, the tests will always require some manual interaction. Whenever manual interaction is required, the test suite will clearly prompt the user and tell them what to do.

One of the hardest parts of tests to automate is control over the VPN applications. Indeed, for many tests, this is the only piece of the puzzle which isn’t automated. We currently have no plans to offer support for automating control of VPN applications to the repository. Adding such support is however supported by the architecture of the suite.

Note that internally, we have implemented full automation of our own application to facilitate continuous integration testing. We don’t plan to make those automation tools available as part of the open source repository.

Setting up the tools

Detailed instructions on how to set up test devices are given in the GitHub repo. We just give an overview here of what to expect when setting up devices.

The test suite is supported on Linux, macOS, and Windows devices (i.e. the Python suite can be installed and directly run on these devices). Mobile devices are also supported by the tools, but the suite itself will never be run directly on these devices.

We recommend using Virtual Machines (VMs) as much as possible, particularly for single device tests. VMs offer several advantages:

• Easy to create and configure new network interfaces
• Allows you to keep your host machine clear of the many third-party software installs that the tests require
• Capturing traffic leaving a VM is simpler than capturing traffic leaving a real, physical device
• Easy to pre-configure snapshots and revert when needed

For serious in-depth testing, we always recommend running the tests on real physical hardware as well, to rule out any potential differences between virtualized and real hardware.

Note that most tests need root (admin) permissions to run, so to simplify things, the whole suite will run as root.

Learn more about leak scenarios the tools test for

The following case studies provide more information about some of the leak scenarios that these tools test for, explaining how they could impact your privacy and security, as well as outlining how to use the relevant tool:

• Case study 1: “Vanilla” leaks
• Case study 2: WebRTC leaks
• Case study 3: Network transition leaks
• Case study 4: Unreachable VPN server leaks
• Case study 5: VPN process crash leaks

Contributing to the tools

We’re more than happy to receive contributions to the tools, whether that be for bug fixes, suggestions for improvements, or entirely new test cases. You can submit contributions via pull requests in GitHub.

Contact us about the tools

You can contact us regarding the tools at leakproofing@expressvpn.com.

Note that the tools are currently in alpha state and will likely change frequently. However, we will endeavor to ensure that the tools are always in a usable state.