Back in the Pleistocene era, I built a website that did PC performance testing. At the end of the test, it would submit a rather large record to be stored in a SQL database. One day, the hosting provider called to tell us that we were experiencing a denial-of-service attack, that the database was stalled, and that the site had become unresponsive. But there was no denial-of-service attack. As it turned out, Leo Laporte was showing his TechTV audience how to use our site. Thousands of people were all performing the same steps at the same time, which amounted to a massive load testing event.
We figured out there was contention for a heavily used database index, one that shouldn't have been there in the first place. We deleted it, but when Leo's show was rebroadcast the database fell over again, this time at a different place. By the third or fourth rebroadcast, the database was tuned up well enough that we were able to start addressing bottlenecks in the business logic, the network topology, and the load balancing.
That trial by fire was my introduction to load testing, and I eventually became good at designing and running planned load tests for all sorts of applications. One thing I learned was how to predict production load quickly. A realistic load test uses large numbers of virtual users with realistic "think times," but that kind of load test often takes hours to run, and a variational study can take a week. You can get a reasonable estimate quickly with intensive testing by using a few virtual users and a very small think time, taking advantage of Little's Law to extrapolate. Intensive tests might only take a few minutes to run, and often you can do a variational study in just a few hours that tells you how to configure your site for a range of loads.
Another important consideration I learned was not to load test too soon.
Too soon, you say?
The received wisdom about multi-user applications is that they have to be built with scaling and performance in mind from day one. That's true, but that doesn't mean you should test under load on day one.
Let's go back to my experience with TechTV. Before the broadcasts, we had done some simple load tests, but that was before that extra index crept into the database. That turn of events was the result of a database system diagnostic aborting prematurely, and leaving its indexes lying around instead of deleting them. It was also before one of the junior programmers "improved" the user experience in a way that loaded down the web servers and before the cookies needed to maintain server affinity throughout a user session were properly configured in the load balancer. Each one of those items degraded the site performance at scale, making the original load test essentially irrelevant.
Knowing then what I know now, I would have done things in the optimal order: First, make it correct. Second, make it fast. Finally, make it scale under load.
What about you? When do you do your load testing?
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
