A customer asked me about load testing recently, which made me realise once again that the terms around both load- and performance testing are horribly overloaded and can mean different things in different contexts.
I thought it would be worth putting a blog post together to explain a few of the different meanings of load- and performance testing.
If I look for “load testing” on the Microsoft website I get presented with the following options.
- Visual Studio Load Testing
- Running Visual Studio Web Tests at scale
- Load Test in VSTS
- Running a Visual Studio Load Test in the cloud
- Load Testing in Azure Portal
- Hitting an Azure resource with a large amount of random traffic
- View and Compare results
- Using the VSTS Load Test integration in Visual Studio to access results in the cloud
While all of these are related to load testing, they only cover a few of the options.
In essence performance testing is an umbrella term for a type of testing where traffic is issued from infrastructure outside of the system under test. The requests issued from the outside are analysed for a range of different metrics like:
- Service Availability
- Response Time
This describes the practice of directing load at a piece of software that will make it load commonly used components into memory. Usually warm up traffic is used to prepare a system for production traffic. The practice ensures that production requests can be served without a deployment-related delay after a product upgrade.
Load testing is performance testing at scale. (A subset of all performance tests are load tests)
When conducting a load test a set of machines will execute a given request pattern in quick succession. Performance metrics will be collected. Most load testing tools average across the performance metrics returned to discover:
- Degradation/Improvement of response time over time
- Dependency execution time over time
- Results over time
A smoke test is usually a quick verification that a system under test is available. (Usually a single call to the system’s endpoint(s))
An availability test is usually conducted from several geographical locations. Technically it is similar to a performance test with the difference that it is run to verify availability of a system from across the globe.
The test is conducted at the same time, from a variety of locations, with a low amount of requests. The purpose is mostly to verify that a system is available and responds as expected.
Availability tests may track further performance metrics such as response times.
Things to consider
Conducting tests from outside a system under test – by nature – regard the system under test as a black box. Many solutions will call several dependencies. (e.g. a database, a third party service, etc.) These dependencies can change the results of a load test. (for example: if we are waiting on a third party service to respond)
Another important consideration is the hardware that the tests are ran from, the network between the testing hardware and the system under test, and the hardware that the system sits on.
The testing hardware needs to support the simulation of the amount of requests required. This can be a challenge when conducting large scale load tests. A variety of cloud services are available to help remove the hardware requirement.
The system under test should be on production-like hardware. If it is not, then the results of the tests may not be fully representative. Penetration tests should be conducted against production hardware and the software under test should be able to respond to the load produced during the test.
The network needs to support the amount of requests issued and the bandwidth required. If it does not, then this may change the results of the performance test. If the network link between the testing hardware and the system under test is metered, additional charges could apply that you will need to plan for.