Functional testing in the cloud: 7 keys to better software

Perhaps the greatest difficulty for functional testers is identifying how to make better software that contributes to the bottom line.

To deliver a great user experience, you must continuously improve the quality and velocity of your testing practices, regardless of the elements you can't control, such as the user's context and technology. Using the cloud for functional testing can help.

But is it right for you? Here are seven things to consider about cloud versus non-cloud functional testing options before making a decision.

StormRunner Functional Testing as a Service (Beta)

1. Lab management

Digital testing requires the setup and maintenance of a modern test lab of browsers and real devices. 

If device lab management or digital lab management is something you want to do yourself, you must strike the right balance of real devices and browsers, taking into consideration the management and acquisition costs as well as the physical and digital security challenges and ease of use.

Digital test labs are extremely inconvenient to manage and should not be a core responsibility. Why add yet another challenge on top of test maintenance? In contrast, cloud-based services can make life easier by:

  • Remotely accessing any platform's—both browsers and physical devices—version and resolution on any version of any operating system.
  • Simulating real-world user scenarios: events, interactions, networks.
  • Running tests in parallel.

That said, there are five things you should consider before purchasing cloud-based functional testing tools:

  • Security (especially with sensitive test data).
  • The potential for increased slowness in test execution, due to such things as load time for virtual machines.
  • Hidden complexity.
  • Lack of stability.
  • The ability to upgrade and update, and to support the latest browsers and real devices (that is, if the vendor provides real devices).

2. Manual testing

Manual testing remains the primary testing practice, but it's typically done for convenience, immediacy, and low cost. Those are all the wrong reasons. You should apply manual testing to get user feedback on specific flows, on the design of your app, and on the overall level of satisfaction.

Manual testing is usually limited to a couple of browsers and devices. As a test engineer, you need to look for convenient ways to ease the exploration and reporting of the user experience, such as automatic screenshot captures, annotations, videos, and device vitals. And once you identify a flow that your manual testers have been repeatedly covering, it's time to transform a manual test into an automated one.

3. Automation practice

Automated functional testing is an essential piece of the equation. It allows you to increase test coverage and time to resolution. These types of tests are good candidates to run in the cloud. 

As an automated-testing engineer, you can:

  • Record your cross-platform scripts from the cloud. Your script skeleton should be fast and easy to create, and you should be able to enhance it with key-press, user interface elements, checkpoints, and other items.
  • Run your existing scripts in the cloud. This means being able to upload your recorded tests and run them directly from your browser. You just select the test in your browser and define the list of environments on which to run it. You don't want to install any plug-ins or local software.
  • Point your local tests to the cloud lab. In this remote-execution scenario, you create and run your automated tests from any continuous integration server or development framework, such as IntelliJ or Visual Studio. But the script actually executes on a browser or mobile device in the cloud.
  • Review your test execution results from the cloud. Whether you are running your tests directly from the cloud or remotely, effective reporting is critical. When each run completes, you need a detailed run report with a summary of the environments against which you ran the test and a breakdown of how each test went. The cherry on the cake is the ability to get a local report that can be saved locally for later review.

4. Testing insights

Why are undetected defects still released sometimes, despite huge investments in automation testing?

Typically, test engineers rely on their own ideas, thoughts, and feelings when prioritizing work. But to comply with expectations of high quality, short release cycles, and reduced budgets, you need to move beyond a simple assessment. For instance, you must move from "Is the app going to function as expected?" to a broader perspective. You need to enhance the accuracy and precision of your testing practices.

One key source of enlightenment comes from production analytics (JavaScript and SDKs) that provide actionable insights into faulty user flows. These analytic tools mine crashes for regression testing and prioritize the browsers and devices on which you need to test.

Applying analytics to improve testing is likely to become one of the most capable and powerful approaches to improving testing as the use of the technology in tools becomes more established.

Any cloud-based functional lab should be able to capture production data and provide instant visibility as to the quality of the tests themselves.

5. Integrations and openness

Whatever cloud-based tool you select, it should comply with your continuous testing pipeline and enhance your ability to design, create, run, maintain, and manage your tests through key integrations.

No vendor offering will be able to meet all of your requirements. But you should expect open APIs, easy integrations, and support among all the tools that your organization uses so that digital teams can leverage previous investments.

6. Licensing model

Organizations are looking for simple, flexible, and easy-to-understand licensing models, regardless of the type of testing, team locations, or duration of projects.

Here are some factors to consider:

  • Simplicity: How is the licensing model structured? All that matters is the number of concurrent users who can access browsers or devices, regardless of the type of testing activity. There should be no hidden options.
  • Flexibility: You should be able to use either a CAPEX (capital expenditure) or OPEX (operating expenditure) budget for your licenses, depending on your needs.
  • Scalability: You should be able to scale up and down according to the duration of any project. The "down" aspect is usually more problematic, so check the fine print.

7. Customer experience

Don't hesitate to do a background check on any cloud vendor, regardless of its size and history. Investigate the overall customer experience, including assessing the user interface, reviewing case studies, contacting support services, reading an online FAQ, or learning details from the partner community.

Get started

The goal of every company, no matter the industry, is to find a way to provide a high-quality experience and high-quality apps for its end users. Testing tools are just one piece of the equation, and cloud-based functional testing tools are no exception. The value derives from the convenience of cloud-based tools and how they fit into your overall pipeline. Taking into account these seven considerations should help you make the right decisions for your teams.

StormRunner Functional Testing as a Service (Beta)
Topics: Quality