When it comes to testing your software, it helps to think like Emperor Palpatine, the character in Star Wars who built an army of clones clad in white armor, ready to do his bidding. Building any serious platform or software package requires pushing, prodding, pinching, and pulling it, just as your users will do. You could recruit an army of real users, but it's much easier to build your own army of virtual testing clones.
The cloud makes this easier than ever. Whether you need one virtual tester or one thousand, cloud services have evolved enough to make it simple to create an army of virtual machines with the push of a button or the execution of a script. Plenty of companies are happy to rent virtual computers by the hour or the minute, so a software tester who wants to simulate the load of 1,000 click-happy users can spin up 1,000 virtual testers for just an hour, all for less than the cost of a fancy lunch.
The cloud is particularly well suited to jobs like this, where demand varies widely. If your server will run 24 hours a day, 7 days a week, 52 weeks a year, building out your own rack of machines is probably going to be cheaper than renting virtual machines in the cloud. But if you just want the test machines for an hour or two, or even for a week, it's cost effective to rent virtual iron in the cloud.
"Why would I want a fast machine?" asks Shay Banon, founder of Elasticsearch, a search and analytics tool. "If need a machine with eight or 12 CPUs, I can just start one in the cloud."
When he's testing his software, Banon spins up virtual testing machines in the cloud, runs the tests, and then shuts them down. The price is reasonable: a loaded 32-core machine with 208 GB of memory would cost thousands of dollars, but renting runs only about $1.50 per hour on Google's Compute Engine.
Software testers can save even more money because they don't need the same guaranteed service that production machines demand. Amazon, for instance, offers a range of discounts for customers willing commit to renting a specific number of hours. It also lets customers bid for "spot instances," an option that can save large amounts if you can work during off-peak hours. Google also offers discounts if you're willing to have your virtual machines be "preempted," a polite way of saying they may shut down your instances during periods of high demand. The same 32-core box with 208 GB of memory will only set you back ninety-five cents per hour if you give Google the option to shut you down. That may be unacceptable for a front-line, customer-facing server, but it's no problem to put aside a test machine for a few hours, or even a few days.
Internal options
Not all companies are ready to move testing outside of their data centers, but they're still mimicking many of the same techniques internally by using dedicated hardware and allocating it to projects as needed.
Assaf Greenberg, a team manager in HP's Performance Center of Excellence, maintains a large, internal cloud that he can allocate to projects. The center has at least 60 physical servers that may be carved up into virtual servers as needed, and they also maintain 20 database servers to store the large data sets required to mimic production loading. Users can allocate these servers to a specific projects to test how a product performs.
"We are loading with 1,000 users," he says of one stress test he was running during our conversation. "We want to see that the new code base is stable for customers."
Setting up internal clouds to be parceled out to developers in house gives the business more control over security while still offering flexibility and balance.
"Some highly regulated organizations are not yet in the cloud," says Theresa Lanowitz, founder and analyst at voke, inc. "In some cases organizations do not want intellectual property in a public cloud for security reasons."
Booting up the cloud
Whether it's in the public or private cloud, launching an army of clones to run your tests is a task that varies dramatically in complexity. Simple applications that run on standard distributions can be spun up by hand using just the standard cloud management tools. But complex applications and scenarios that require more than a handful of virtual testing machines are usually better accomplished with customized software that automates much of the routine. It all depends upon the requirements of the test plan.
The basic tests don't require much preparation or software. The various cloud platforms offer web-based interfaces that can spin up a virtual machine in minutes. You can choose the version of the operating system—Ubuntu is the most common—and the size. After you update the operating system, you install your software and start the tests.
The basic interface lets you save an image of your fully configured virtual machine, so you only need to install the software once. Often, this is enough for simple solutions, such as creating an army of browsers that repeatedly pound on a web server. I've personally written some scripts that load web pages repeatedly just to do load testing. If I save an image of this machine with the scripts installed, it takes me only a few minutes to spin up a handful of machines—that generates plenty of simulated load for a server.
But building the test constellation by hand only works with simple test cases. If you're constantly installing new versions of the software with each cycle, it helps to use container software, such as Docker. All cloud providers offer at least some measure of support that simplifies the process of getting containers running on a virtual machine instance.
If you're building large clusters of Docker containers, you can use Google's open source Kubernetes tool to juggle them. The search engine company developed Kubernetes to quickly build out clusters defined by Docker containers. You configure a cluster by defining how the containers should be grouped as pods, and then use Kubernetes to pack them into nodes.
Developers who need to test large clusters can use Kubernetes to start them up. You can use it to start new virtual machines if necessary or simply to add a container to an existing virtual machine. Kubernetes handles all the work of spinning up the environment.
Automating configuration
If you're creating more than one instance, the idea of automating is enticing; if you're creating more than a half dozen, it's essential. Many tools are now available to simplify the process of configuring machines. People commonly use these tools in production environments to push new versions of the software into use, but you can also use them during development.
The two most common open source products are Puppet and Chef. These competing systems offer similar architectures and have similar goals but reflect different philosophies. Both emerged as enterprises began embracing systems with clusters of computers that needed to be kept in sync. Both take a description of the configuration and then apply it remotely to all the computers in the cluster. This description, usually called a "manifest," is converted into a sequence of lower-level commands like "apt-get" or "rpm," depending on the underlying operating system.
The philosophical differences between Puppet and Chef lie in the languages used to spell out the configuration of the computer. The Puppet manifest is said to be a model that merely declares what the final configuration should contain. Puppet translates this into the list of lower-level commands. Chef, on the other hand, uses "recipes" that spell out the commands and the order in which they will execute. Both have devoted followings and passionate defenders, and both end up delivering pretty much the same service. It's worth noting that there are also competing packages out there that you might consider that don't get as much attention.
Using a configuration management system is a good practice, even if you're starting with one machine. While the hassle of writing a script may be lost if you're only building one or two instances, it can be run repeatedly. If you decide to scale up your test by adding more, you can do it quickly.
Beyond configuration: Virtual labs
Configuration management systems can automate much of the hassle of setting up a virtual test lab, but they don't do everything. Coordinating the start of the machines and triggering the update are out of scope. This is prompting a new, higher-level solution that some companies are packaging as environment as a service (EAAS), or virtual labs.
These tools aim to add all the configuration options that aren't easy to create with basic systems. Brad Schick, CTO at virtual lab vendor Skytap, says his company's virtual lab isn't a provisioning tool. "Skytap is a service that provides environments in which to run provisioning tools, along with the rest of a complex application stack." In other words, the virtual lab can trigger programs like Puppet or Chef along the way. The product also embraces other tools, such as Jenkins, Team Foundation Server (TFS), and UrbanCode. They're all just another set of jobs that are triggered along the way, he says.
Skytap lets developers and testers juggle "environments"—collections of instructions, virtual machine images, and configuration files. Individual machines can be clones of golden images or built out of Puppet commands. You link them together with a network infrastructure that can include load balancers, encrypted VPN tunnels, and other complex topologies to accurately test the applications when network loads and lag can change performance.
"Once this 'golden template' environment is provisioned in Skytap, it only takes them seconds to stamp out and share as many exact copies as they want," says Schick.
Kubisys offers a similar service for developers working with a Windows environment, but with a twist. It will spin up servers and mirror existing hardware topologies, but it can also install a read-only tunnel between production databases and test environments so developers can test new software with real data, without impacting the production database.
"We take a snapshot and that snapshot stays on the host server and then from our appliance, we read only the data that's required for the user test, " says Jeff Martinik, senior VP of sales at Kubisys.
"You know when you test, you don't need all the data, but for you to have a good test environment, you need to copy all of the data," he explains. "So we leave it there. We read only. There's no impact on your production."
Lanowitz at voke says systems like these do more than just simplify the configuration process. They also standardize it and ensure that testers can build machines that closely mimic the production servers. Teams can use the same configuration scripts for both testing and deployment, ensuring consistency.
"Virtual and cloud-based labs provide testers with an environment as close to production as possible for testing," she says. "With one click, developers are able to immediately reproduce a defect, since developers and testers are now working in an environment as close to production as possible."
A scientific approach to testing
Choosing the right level of testing for your product is an art, but once you've decided just how much of a load you need to test, the process becomes more of a science. Using the right amount of automation not only simplifies the job of organizing the tests, but it adds another level of precision by ensuring that the test environment is consistent. Tools like these may be designed to automate the scut work of configuring dozens of machines, but they also provide a structure and regulation to ensure that the tests are scientific and reproducible.
"Every night we are running a regression test," says HP's Greenberg of his group's development test lab. "At the end of the test, we compare the results to the previous run in case we have a regression between the builds."
Each morning the developers and testers receive an email summarizing the success or failure from the tests that ran for about five hours the previous evening. If there were problems, they can drill down and see which transactions triggered the failure.
And if they need to replicate the results again? It's easy, because everything is automated, says Greenberg. You only need to push a button to run the experiment as often as you like.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
