Table of Contents
Overview
Performance engineering is about far more than just performance testing. It's about achieving consistent, predictable system performance under load, with changing conditions, and even during a partial system failure.
To accomplish this, performance engineers engage in capacity planning, modeling and simulation, development, performance testing, platform engineering, and production monitoring.
The body of knowledge performance engineers need includes system administration, coding practices, networking, CPU and memory management, the theory of constraints, testing, and traditional debugging skills. More recently, cloud provisioning, DevOps skills, microservices, and non-relational databases have become relevant as well, said Vicky Giavelli, director of product management for performance engineering, Micro Focus
"You'll need a complete set of tools to engineer performance; the problem is just too broad."
—Vicky Giavelli
In other words, any systems in the cloud or the data center could become part of a performance engineering effort.
Why read this report?
This guide covers the most important benefits of performance engineering, the key features and tools available, and how to select the right tools for the job.
The wrong tool can waste time and money, but the wrong selection process can waste twice as much of both. To help you buy the correct tool and use your time judiciously, this report defines the current state of the tools, including what they offer. You'll also learn which criteria to consider for each tool category.
The RFP section covers the questions you should ask performance engineering tool vendors. These are not simple, cut-and-paste, hands-off questions. Instead, to get the most relevant answers, those making the technical selection should work closely with the engineering team and the vendor to select the right questions and provide the right background.
Performance engineering can be challenging, but it does not have to be overwhelming, nor does it need to break budgets or timelines. Understand the problem, model it, test it, evaluate it, improve it, and choose tools to monitor and improve production.
That's how you take the guesswork out of performance engineering.
The state of today's performance engineering tools and techniques
The performance engineering discipline is in transition from the old waterfall approach, where performance testing was done by a single expert (if at all), and production monitoring was handled by the IT operations group.
Now that's in flux, with the work distributed within engineering teams. Thanks to DevOps, those same teams are adding production deploy and production support to their workloads.
Performance engineering as a role has diminished
On one hand, "everybody pays lip service to performance, and indeed the role of performance and efficiency grows," said Alexander Podelko, a consulting performance engineer at MongoDB. "We are spending more on it."
But on the other hand, he added, "the number of people with 'performance' or 'capacity' in their titles is shrinking," with the responsibility spread between site reliability engineers (SREs), developers, software developers in test (SDET), and so on. "Yet no one has specific responsibility for the subject."
This extra responsibility can lead to waste, with elements of the test picture falling through the cracks. Without anyone to coordinate the process of fixing problems, teams can develop gigabytes of data without any relevant insights, or insights without any actual corrective activity.
In the worst case, this can push teams backwards, from performance testing into production firefighting, also known as "code and fix" behavior.
Current tools can be difficult to integrate
Most performance tools were built for performance testers, to serve a single function. They may be difficult to integrate. And when they do work together, the architecture is frequently closed, encouraging a single-stack, instead of best-of-breed, approach.
Worse, because the responsibility is now distributed to other roles on the team, those people have to leave their work environment and set up an entirely new workspace to do performance. Developers who are offered tools that are not tied to the style or scope of their core work are likely to abandon those tools.
One result: Tools that report on performance are viewed in isolation. Test results are viewed by a "test group," application performance monitoring (APM) data and logs are viewed by operations, and front-end performance, if it is measured at all, is measured by a design or front-end specialist.
There is no shared analysis, especially for complex systems developed by teams in different locations and time zones.
Performance testing has a high startup cost
An excellent load test will have multiple users performing somewhat random operations, perhaps from different locations. The test will need to include the accounts, the possible workflows, perhaps logic to track state, as well as meaningful measures on the test data.
That is, you need all the test data—the average (mean), typical experience (median) load speeds including the front end, as well as perhaps the experience for the slowest 10% or 25%.
Setting all of that up, including creating the test environments, takes a significant amount of time and money. But even more than the tool cost, the human time required to do the work invariably drains the entire delivery team, not just the tester. Physical hardware, which is more often the case than not, can be expensive, especially in large organizations with multiple teams that do not coordinate or reuse resources.
Leandro Melendez, a consulting performance tester at Qualitest, warns that he still sees many organizations with a physical test environment that's a miniature version of production. This saves money but guarantees that the test environment is not a realistic model of production. Since most larger organizations have no way to share performance test resources, teams have to reinvent the wheel, creating more cost and delays.
Performance testing alone remains a bottleneck
Given that the setup of performance testing is expensive and slow, it will inevitably push out the schedule. Vicky Giavelli, director of product management for performance engineering at Micro Focus, said that leads to either a delayed release, or, more likely, to an attitude of "catching it in the next release" or "we'll test it later." This, in turn, inevitably leads to "testing never" and firefighting in production.
Key takeaways
Large organizations have DevOps teams that tend to focus on infrastructure and setup, leaving testing behind. Without infrastructure, testing falls even further behind. That bottleneck could eventually lead teams to ignore performance testing itself, resulting in production problems that could cost customers, sales, and the expense of fixing the problem.
But the news is not entirely bad. Pushing for teams that "own" large parts of the process, from product design to support, creates an opportunity to extend performance engineering throughout all of the delivery lifecycle, from concept to cash.
Centers of excellence and communities of practice can push down information to the team about how to test. As tools evolve to become more open and better integrated, to support the cloud, and to work for different specialties within software development, performance engineering has a chance to become a team sport, with a lighter burden on each person.
What to look for when building a performance engineering toolbox
The simplest performance testing toolbox includes a load test generator, hardware to generate that load, and some way to monitor performance. Those tools tell the tester how the system is performing, but they don't help fix it. They also don't focus on general performance metrics that are unrelated to load, such as unitary, single user, web performance, and so on.
Performance engineering requires a broader approach and perspective. It includes prevention, modeling, analysis, and repair.
Figure 1. Performance testing is the least-implemented type of testing within a continuous delivery pipeline, likely due to a lack of proper tools to perform the work. Source: CapGemini's 2020 Continuous Testing report
Here are the types of tools to use for performance engineering.
Monitoring
This isn't really a testing tool, but all the load and simulations in the world don't mean anything without data on system performance. Sometimes called application performance monitoring (APM), these tools should provide drill-down dashboards along with mean, medium, top 25%, bottom 25%, and other metrics on any operation.
A good data dashboard can receive information from anywhere, making it possible to report synthetic transaction results, and logs for delays for every subsystem, along with cloud usage and CPU, memory, and disk use statistics.
This combination for test and production makes finding problems, developing hypotheses, and conducting experiments an engineering exercise, instead of guesswork.
For the purposes of this report, a monitoring tool stores the data, reports on it, and either gathers it or has "listeners" that forward the information to it. An APM tool will have additional API hooks into the systems themselves to trigger tasks in the system, perform synthetic transactions, or sometimes inject an audit trail into the code itself.
Load generation tools
A performance test tool usually means a load simulator, scripts and data to send to the load simulator, and a view of results. These tools predate the web and used to run through the user interface. As mobile and microservices have become more popular and software has become incremental, the ability to record production queries and replay them is increasingly popular.
Front-end performance tools
These tools analyze what is downloaded, how long it takes, how large the download is, and how that impacts page load. A simple load fix might be to swap out a high-resolution image for a medium-resolution one, or one with better compression. Note that many vendors include front-end performance and load generation in a single tool.
Service virtualization/mocking
Some teams directly support only one subset of the system, or a handful of microservices, and those services may be dependent on a wider part of the system. By recording how the other systems interact and then playing it back, it is possible to test just the subsystems a team builds in isolation.
That can lead to faster testing and easier debugging. The term for these lightweight, pre-programmed services with a limited number of requests/responses is "service virtualization."
Network virtualization
In many cases, the delay is in the network infrastructure outside of the data center. Network virtualization allows a test to simulate a 3G/4G/5G connection, perhaps to a different continent, without the expense of having to fly someone to Europe to use an older phone model. Armed with this data, programmers can reduce the size and complexity of downloads, improving performance.
Static code analyzers
While these aren’t typically included in a list of performance engineering tools, they're good for finding memory leaks. If your software has memory leaks, which cause delays or errors, a code analyzer could prevent them from getting into your code in the first place.
Code profiling tools
A different approach to performance is looking at how long each line of code takes to execute, and how many times it is executed.
Code profiling tools show where the code is spending its time, providing opportunities for improvement. (Of course, it is possible that the network is slow.) This is essentially a dynamic version of code analysis, or analysis as the code runs.
Query and database optimizers
A single line of code that takes too long might be calling out to a database. Query optimizers change the way a query or command is structured to make it run faster. Database performance tuning is an entire discipline, but it certainly fits within performance engineering.
Build and provisioning tools
These tools allow self-service and the creation of test environments on demand. For a deeper understanding of why these tools are important, read how Bernie Berger, an IT and quality coach at EPAM Solutions, described the absence of build tools and test environments in his "Day in the Life of a Software Tester" series of articles.
In Berger's examples, the tester might spend four days on setup and configuration to perform a one-hour test. Provisioning tools create a data environment, and possibly seed a test database, so that performance tests can be executed quickly.
Chaos engineering
These tools work to pull down subsystems intentionally to see if the architecture has the necessary redundancy to survive failure. Chaos tools plan for failure, simulate it (often in production), and require layers of redundancy to create highly available, resilient architectures.
Now pull it all together
There are quite a few types of tools to consider. Fortunately, you probably only need a subset of these tools, and you might want to implement them in a specific order. Given this information, most technical groups should be able to explain what is missing from their current tool lineup, and how any new tools will fit with existing systems to create a performance engineering culture.
The challenge will be deciding who will do the work, how they will find the time to do the work, and the right tool to do the work.
Once you’ve completed these steps you're ready to create an RFP and move forward with a proof of concept to help narrow down your tool choices.
How to create an RFP for performance engineering tools
Once you understand the landscape of tools, the challenges in your organization, and the relevant features, it is time to get down to tool selection. Part of that process might be creating a request for proposal, or RFP.
The RFP can be an important part of the process of limiting a large group of tools to the select few. That said, Qualitest's Melendez warns that the process can remove all context from the conversation. Leaving aside the vendor, making assumptions about what terms such as "build provisioning" mean for this particular client or other missteps can lead to the wrong decisions.
Figure 2: Accurate performance insights of real users are the top challenge for continuous testing. Source: Capgemini's Continuous Testing Survey 2020
The RFP process should look more like a conversation, with bidirectional feedback and explanations.
Describing your company and performance-engineering problems
Most performance engineering tools solve a specific problem, address a specific need, and work in a single area of performance engineering. Even companies that offer an entire integrated suite of tools, designed to address multiple problems within performance engineering, present those as a suite, not a single tool.
Think of them like puzzle pieces that fit together to craft a solution.
You want to know how the pieces will fit. By describing your organization, the existing performance culture and approach, the desired state, and the gaps you're trying to address, you allow the vendor to answer questions about how its piece will help solve the puzzle. The vendor can also suggest anything else that may be needed, based on its experiences with other customers.
This section consists of three components: goals and objectives, resources, and infrastructure.
"The tool may meet all the requirements, and actually be able to solve our problems. But do our people have the skills and capability to use it as intended? Moreover, will they?"
—James Pulley, test practice manager at TEKSystems
The outline below is a guide; you'll want to tweak it to fit your group and the type of solution you're seeking. Monitoring tools should have a very different set of questions than performance testing tools.
Statement of goals and objectives
Start with the current state of performance engineering. The temptation here is to overestimate the existing state, to talk about the way things should be. Have a technical person, who can set aside bias, write about the true state of what is really happening.
Follow with the end-state goal for performance engineering, any gaps that exist, ideal timelines, and how success will be measured. If you can, include information about budget, existing slow performance, and the effect that improved performance could have on the business.
Statement of resources
This is about who will be doing the performance engineering work—whether it will be an additional duty, whether you will create a specific role or use consultants, the number of people who will be involved, their skill sets, and what group they will report to.
You may also include a high-level description of the IT department, along with more detail on the system administration, database administrators, programmers, team composition, development model, continuous integration approach, DevOps, and other areas that part of performance engineering.
Infrastructure
This section describes the hardware and software elements of your IT environment relevant to performance. That includes the operating system, hardware and database, and the public or private cloud infrastructure and chipset. An all-Microsoft environment and a Linux-heavy environment may lead to very different solutions, as might a Java-, .NET-, or Kubernetes/Docker-heavy group or a particular brand of enterprise service bus.
Include any security issues or relevant regulations. For example, specify if all the software needs to run inside the data center, or if the system needs to be compliant with Sarbanes-Oxley, Dodd Frank, HIPAA, PCI, FDA, or other regulations.
Soliciting vendor information
Once you've created the infrastructure section of your RFP, it's time to learn about the vendor and its tools. Consider asking the questions below in the vendor information section of your RFP.
- Describe your company, history, and solution focus.
- What is your financial situation? Provide at least two years of financial data.
- Who is the primary contact for sales?
- Who is the primary contact for technical questions and support during a trial period?
- Do you have a list of reference customers we can speak to? If so, please list them along with contact information. Links to case studies are not a replacement, but can be listed here if available.
Products and capabilities
This section allows the vendor to explain its product, approach, and any competitive advantage it has. Ideally these answers demonstrate that the vendor's representatives actually read the previous section and understand how their products will fit into your IT picture.
- List your product or products and the primary category each falls into: capacity planning, modeling and simulation, development and DevOps, performance testing (include web performance and front-end performance), platform engineering, production monitoring.
- What technologies and API protocols do you support?
- What cloud and virtualization technologies do you support?
- What skills are required to use this tool?
- How much time and additional training will we need to be effective?
- What devices and platforms does the tool support?
- Do you provide extensions to support performance engineering for commercial products (ERP, CRM, HR systems, and so on)?
- How does your product work with other products—via APIs, or other means? What other products does your software typically interact with? What open-source products?
- What continuous integration or continuous delivery tools do you support/integrate with?
- Describe how to customize or extend your product.
- What types of reporting and analytics does your product provide?
- What unique features or innovations does your product suite have compared to the competition? Whom do you consider to be your competition?
- Does your product, or product suite, work better as a suite, or as part of a particular stack of technologies?
Deployment and support protocols
- Is your product or service on premises, a cloud service, or part of a hybrid solution?
- Does a typical installation involve consulting or training? Is it via a third party, or do you provide it?
- How often are new versions released, and how does that impact the licensing model?
- Do you offer support? What are the terms of your support contracts? (It might be helpful to ask for support during the process and see how that goes.)
- What training options do you offer, and at what prices?
Describe the licensing and pricing model in a separate document. This should include fees for virtual users (for load testing), real users, if the license can float between computers, cloud hosting costs, and so on.
Once the RFP is drafted
Pulley, the test practice manager at TEKSystems, provides these examples of problems the RFP should help answer:
(1) Does the tool in question have the ability to interrogate all my systems to understand how resources are being used? (These include operating systems, guest operating systems, and services web server, app server, database server, and so on.)
(2) Does it work in my environment? Does it run on Macs or Linux or Windows or that embedded thing that Bob has running a robot? If not, then it may be a perfectly capable tool but it can't be used in my environment.
(3) Can this tool be used successfully by the intended users of the tool? The scripting language is Erlang. No one knows Erlang. Heck, the group is full of manual testers, and almost no one in the group has programming as a background. All performance testing tools assume a robust level of architectural knowledge and programming acumen. You cannot diagnose issues in code and architecture without a foundational command of the same.
Once you get responses to the RFP, take a hard look to see if the vendor really answered your questions or engaged in marketing-speak, and if it seems like it will really work to resolve the performance engineering issues you require.
After the RFP has passed muster with you and your team, send it out for review to any others who will be using the tool. Then it may be time to create your short list and start talking to vendors. Good luck!
