Performance engineering is the discipline of determining the performance requirements of a system, then designing, evaluating, and improving that system to accomplish those requirements. Before you review the latest performance engineering techniques or begin the process of selecting the right tools, you should understand these basics.
The essential elements of performance engineering include capacity planning, modeling and simulation, development, performance testing, platform engineering, and production monitoring. That definition is largely based on the work of James Pulley, practice manager for release and test at TEKsystems and co-host of the PerfBytes podcast.
Figure 1. Core elements of performance engineering. Source: James Pulley
While there's general agreement on this definition, opinions on how to accomplish it vary. A major issue with performance engineering is proving its benefit to the business, said Andreas Grabner, a DevOps advocate at Dynatrace. Anyone involved with systems performance knows the advantages, but they are not always obvious to the business people who approve the budget.
Consider Amazon's claim that every extra 100 milliseconds of latency costs 1% in sales, and Google's claim that an extra half-second in search page generation drops traffic by 20%. And the TABB Group estimates that if a broker's electronic trading platform is 5 milliseconds behind the competition, it can lose 1% of revenue.
These figures estimate the cost of only a small delay. An extended delay, even a few hours of downtime, could mean loss of reputation that ripples out for months or years. Trying to mitigate those risks with performance testing alone creates new costs and delays. Worse, that limited approach fails to integrate into a modern DevOps world.
While performance engineering evolved from performance testing, it now encompasses much more. Here's what you need to know to get started.
Performance testing's beginnings
Classic performance testing took a "build and test" approach. Fred Brooks called that "build it twice" in his landmark book Mythical Man-Month: Essays on Software Engineering, perhaps the oldest work about software engineering to remain in print. Brooks said, "Plan to throw one away; you will, anyhow."
In days of yore, traditional performance testing might be done once, just prior to the software's release, and involved a fair bit of tweaking. If modeling happened in any formal sense, it was what performance-testing guru Scott Barber called "with crayons," drawing a flow diagram of the website and guessing about user actions.
The thinking was that performance testing could not happen until the entire system was built and assembled. That made performance testing part of a waterfall process, instead of being done an incremental piece at a time.
Software testing consultant Michael Bolton puts it succinctly: The core issues in software are less often testing, and more often fixing.
Performance engineering both fixes and prevents problems. The goal is for performance problems that appear in any change to be identified as part of that change, almost immediately, or within minutes of it moving to production.
Modern performance engineering
Especially given the adoption of DevOps, you will need to understand about moving testing "left" and "right." The term shift left means moving conversations about quality, requirements, and how things will be tested to places that are earlier in the process—and taking appropriate action. Shifting right involves releasing the code earlier, perhaps to a small subset of the user population, a technique sometimes called "testing in production."
Several years ago Goranka Bjedov, then a capacity engineer at Facebook, pointed out that her company was shifting performance right. The company could push a change to a specific cluster and monitor that cluster's performance. Performance engineering combines shift-left and shift-right techniques to make performance built in, or engineered, into the process.
The modern approach takes the disciplines listed above and applies technologies to the problem. Performance engineering includes continuous integration, continuous delivery, virtualization, and cloud-based deployments. Production monitoring is more than application uptime and response time, and includes application performance monitoring (APM).
According to Gartner, APM includes front-end response/load monitoring, application discovery/tracing, and analytics. Once a change is launched, APM provides the tools to see how that change is performing and allows you to roll it back or adjust if there are problems. Performance engineering can also include canary testing, to deploy a change to a limited population and test in production.
Leandro Melendez, a consulting performance tester at Qualitest, suggests cloud-based delivery as the "secret weapon on QA modernization." That means the ability to create a test environment on demand for any given change, including all the infrastructure automation needed. Once that infrastructure is in place, continuous performance evaluation becomes possible.
Continuous performance evaluation
Compared to waterfall development, which built, tested, and deployed everything, the DevOps approach builds one piece at a time, tests it, deploys just that piece, and monitors for success. That makes performance engineering and evaluation part of the continuous integration (CI) pipeline.
Performance testing tools need to generate demand, run, and publish results in a test environment that is created on demand, as well as load realistic test data. To do that, these tools need to integrate with a wider set of infrastructure, creating output that can be consumed by the CI pipeline and reported on. The tools also need to provide enough information to allow code to be debugged after a failing run.
Qualitest's Melendez suggests that, in addition to test results (response times), you may want to review CPU, disk, memory, and network utilization, along with trends. In a complex environment, delay, tracing, and debugging information will help with the next step: remediation.
The same tools can help with monitoring production, to respond to changes in performance due to changes in demand, customer use patterns, or code rollouts with unexpected use cases. Some monitoring tools can capture production customer behavior, to create models for testing, or even make real production data anonymous to create accurate performance and load test scenarios.
Make the right investment
Finally, there should be a sense of realism, or business value, about this effort. Alex Podelko, a consulting performance engineer at MongoDB, said performance engineering mitigates risk, but cannot prevent it.
Time and effort spent on performance engineering should result in much more time and money saved than invested. That means there is a sweet spot where you'll hit the maximum ROI. To do that, you'll have to choose the right tools and techniques.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
