It used to be OK to spend a long time on application deliveries. Enterprises might spend anywhere from 18 to 24 months on a major release. In today's automation-oriented technology environment, however, that's completely unacceptable. What's considered cutting-edge technology today will be outdated or even redundant 18 months down the line. Customers expect releases at least once per month, and if you don't deliver, they'll go to the nearest competitor who does.
The requirements for modern application delivery have exploded along three vectors: frequency, diversity, and openness.
Ten years ago, if you supported one browser or a single class of mobile devices, 90 percent of your target market was covered. Today, Internet Explorer, Chrome, and Firefox all have significant market share (not to mention a host of other niche browsers that shouldn't be ignored), and the number and range of mobile devices and combinations of operating systems keep growing exponentially. To complicate things further, many companies develop solutions that need to run both on-premise and in the cloud.
If that's not enough, your product needs to expose APIs so that more and more systems can be integrated. These are common expectations that make customers sensitive to defects in your product. If one of your APIs breaks, their integrated systems break.
Looking at the three-dimensional testing matrix these vectors create can be dizzying, but there's a solution: automation.
Automation is what fires modern application delivery methodologies of continuous...well...everything. Taken to the extreme, in a fully automated software delivery pipeline such as DevOps, any piece of code that is committed is automatically run through a build process (continuous integration), which includes a barrage of automated tests (continuous testing). If those tests pass, the build is automatically promoted through the pipeline (continuous delivery) until eventually, it's made available to consumers. While most organizations don't quite take it to that extent, these automated processes are what enable organizations to overcome the impossible vector of variables to which they must cater.
Be strategic, not tactical
The concept of automation in software isn't new. Automation tools have been around for years, but successfully introducing automation into your organization will take more than just selecting this tool or that scripting language. You must have a conscious strategy backed by your organization's leaders. You'll need to allocate adequate resources and treat automation as a critical line of development, complete with priorities and clear backlog and release cycles. You'll need to measure progress on an ongoing basis, with concrete metrics that show whether you're achieving your goals or not. You should handle automation as a full-fledged software project with several contributors, not just a token expert on whom the whole project depends. If properly nurtured, your organization's automation infrastructure should mature and grow to be robust, maintainable, and scalable.
Don't just adopt tools. Identify and treat pain points
Implementing automation properly requires a big investment. If you don't quickly show a great return on investment, your project will be killed. Don't let any particular tool be your driver. Instead, look for pain points in your organization, and don't start with a huge project. Go for something small, clearly defined, and well scoped. What do you want to achieve? Do you want to replace manual efforts? Quick feedback on new builds? Better platform coverage? First decide on the problem you're going to tackle, and then choose the best tool for the job. Last, make sure you can demonstrate the value of your efforts with quantifiable results. For example, you could show how many workdays of release testing are saved, how many builds can be tested per week, or how you can increase platform coverage. There are many ways to measure effectiveness. Choose the one that demonstrates improvement on the most painful point in your organization's software delivery pipeline.
Get everyone involved
Keeping any knowledge domain in the hands of one expert is risky. People change jobs, get sick, or move on to other projects. You do need a clear owner for every automation project—someone who will keep it moving toward clearly defined goals—but everyone else in your organization should also be involved. No single person should be responsible for developing and maintaining the whole automation infrastructure. If necessary, spend some budget resources to train your team members so that everyone can at least write a simple script, run tests, and analyze the results. Your experts can handle the more complex tasks of writing frameworks, designing your automation architecture, or even mentoring other team members.
Keep it robust and well maintained
Frequent false positives make people lose confidence. If your automation code fails a build because of a bug in the script (as opposed to a bug in the product), it'll take a lot to rebuild confidence in the process. Start with small, efficient tests and build your infrastructure slowly so you can keep it well maintained.
Here are a few things you can do to help:
Make sure your scripts create detailed logs. This way, when something goes wrong it will be easier to debug.
Use screen shots when running automation on a user interface (UI). You don't have to record the whole session if storage is at a premium, just taking screen shots when errors occur is helpful.
Make sure your developers can access and debug the testing environment if necessary. If you run tests on a server with limited access, your developers will be stuck if something happens.
Have a process in place to handle errors in your automation flow quickly and efficiently. Who gets notified by email? How is it handled? Who is responsible for managing the error? Is it development, DevOps, or QA? Make sure this is an integral part of the process on your build pipeline so that it doesn't result in a discussion every time an error happens.
Divide automation efforts between UI and API
In the early days, a lot of effort went into creating automation scripts to test UIs, but it wasn't effective. Even then, product UIs changed so quickly that it became challenging to maintain testing scripts. More recently, the focus has moved to application programming interfaces (APIs). Since APIs are instrumental in separating application layers, testing an API directly makes it easier to isolate errors in the back end. UI testing has not been completely abandoned, though. You can still effectively test your UI for mature legacy systems that may not change frequently. In newer systems, the main UI flow may also be stable enough to justify developing scripts to test it.
Automation is the only way to support agile
Agile development methodologies are an integral part of modern application delivery. Developers and QA professionals must collaborate closely to provide quick feedback and shorten development cycles, which means that builds may be delivered for testing on a daily basis or even several times a day. Every build must undergo minimal sanity testing to ensure integrity before being passed on for more intensive QA, and it's impossible to keep up with that kind of pace without an automation safety net. While the benefits of a stable build aren't always measured using the classic metrics of workdays saved or number of defects found, they're key to the organization's efficiency and ability to quickly deliver high quality products. If your organization is already agile, make sure automation is included in your definition of "done," just like any other functional requirement, and you'll move another step closer to DevOps.
The benefits of automation are clear: it reduces the time and effort you need to spend on QA while providing quick feedback and expanding code coverage, ultimately resulting in a better product. However, enjoying these benefits takes a significant amount of organizational resources. Simply hiring a part-time student to do the work manually won't cut it; automation must be part of the organization's DNA. While this can be costly at first, the ROI comes quickly enough. If executed properly, introducing automation into your organization is worth the investment.
Image source: Flickr
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
