The goal of load testing is to find problems before they impact users. The more realistic your simulation, the more likely you'll catch bottlenecks that lead to a bad user experience.
While many variables affect accuracy, the number of concurrent virtual users is one of the most important. Ideally, you could test with as many virtual users as you need. In practice, this may be too expensive because load testing software is priced on the number of concurrent virtual users.
So how accurate do you need your performance testing to be? How many concurrent users is "good enough"?
How many concurrent virtual users do I need?
Figuring out how many concurrent users you need isn't always straightforward.
For new sites that haven't yet launched, anticipating real user traffic can be difficult. In this case, a soft launch (an internal/private beta) is invaluable to learn actual user behavior and get some indication of traffic amounts and usage patterns. If that's not possible, looking at comparable sites can also be helpful. It's almost always better to have a setup that is initially overkill (since you can turn down resources as needed) than vice versa.
For existing sites, you usually take existing traffic as a baseline and double it, triple it, or multiply it by a larger factor. This is usually a safe bet.
Before continuing, we need to make an important differentiation. If you're testing a website, then the concept of concurrent users, each with their own set of cookies and session data, is indeed applicable. On the other hand, if you're testing a stateless REST API, the concept of concurrent users might not be applicable, and all you really care about is requests per second. This article discusses the scenario where you do indeed care about the number of concurrent users, and not just requests per second.
Concurrent virtual user calculation
Now let's look at some numbers. Here's the basic calculation:
(concurrent users) x (requests per user per minute) = total requests per minute
For example, if you run a load test with 10,000 virtual users, each making a request every 20 seconds (3 requests per minute), then you're making 30,000 requests per minute, which equals 500 requests per second.
As you can see in the calculation above, you can decrease the number of users and increase the number of requests per minute per user and still have the same requests per minute. For example, here are a few scenarios that all generate 30,000 requests per minute:
(10,000) x (3) = 30,000
(5,000) x (6) = 30,000
(1,000) x (30) = 30,000
(10) x (3,000) = 30,000
Considering that cost is often proportional to the number of concurrent virtual users, the question arises: Instead of testing with 10,000 virtual users and 3 requests per minute, can you test with fewer users and more requests per second and get the same test results? After all, you're hitting the back end with the same total number of requests per minute.
The answer? It depends.
How to balance virtual users versus requests per user
Performance testing is done to identify bottlenecks in your system. These bottlenecks could be memory-, CPU-, or network-related. The bottleneck itself could be anywhere in your app code, database, or caching mechanisms.
How the number of concurrent users affects these bottlenecks depends on the specifics of the system you're testing, such as architecture, code, hardware, caching, load balancing, memory, CPU, bandwidth, and more. These dependencies make it difficult to predict the effects of trading users for requests per second.
Ideally, you could run a few tests with various levels of virtual users, keeping the number of total requests per minute the same, and see if the actual results differ. This will give you some indication of how moving the virtual-users level affects results, though every possible scenario would need to be tested and this option isn't always viable.
Still, considering the architecture of most websites and web apps, testing with fewer concurrent users may produce overly optimistic results. In other words, false positives are less likely than false negatives.
Let's return to our example with 30,000 requests per minute. If you identify a bottleneck with 5,000 virtual users at six requests per second, it's unlikely that this is a false positive. Chances are you'll also see a bottleneck when testing 10,000 virtual users at three requests per second.
On the other hand, if testing with 5,000 virtual users at six requests per second doesn't identify any bottlenecks, you might have a false negative. Testing with 10,000 virtual users at three requests per second may reveal that you do have a bottleneck after all.
There are two common reasons you'll see false negatives:
- Storing server-side data per user session. This is almost always the case with e-commerce and web apps. This means that fewer concurrent users = less memory usage. Less memory usage = less bottlenecks.
- Use of some caching mechanisms. This means fewer unique users = more cache hits.
More cache hits = fewer bottlenecks.
How many real users do you have?
I often hear of companies that want to run a load test with a million virtual users. Upon further investigation, I discover that their website gets one million unique visitors per day (or week, or month) so they think they need to run a load test with one million concurrent visitors. Obviously, this isn't the case.
To put this into context, when load testing vendors talk about concurrent users or virtual users, they're usually referring to two aspects:
- Each user has its own unique cookies, session data, internal variables, and so on.
- All users can make a request at the same time (concurrent).
You should be able to ask your dev or web analytics team how many concurrent visitors you're really getting.
Proceed with caution
When load testing, it's always best to test with an accurate number of virtual users. However, you can often reduce the number of virtual users and still get accurate results, though you can't know for sure and are taking a risk.
Understanding the architecture of your website or web app is critical to making the right call. Good communication with your dev and web analytics teams is a smart place to start.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
