One of the most critical questions test engineers face when starting a new test automation project is what tools to use. Nowadays, many automatically consider only open-source tools. But is that always the correct choice?
Based on my experience, it isn't. The pendulum has swung from vendor to open source, but as is usually the case, the truth is found somewhere in the middle. Open-source testing tools are awesome, but there are many instances when they just aren’t enough. Here's when to start looking for a commercial tool.
When you're testing more than one technology
When choosing a test tool, it’s wise to first take a step back and clearly identify your testing needs. Open source might satisfy them, but a commercial tool may be better; just be sure you’re making an informed decision before choosing one over the other.
To do that, list all of the different technologies your organization will have to automate against. For larger companies, having one test tool that you can use across all teams and divisions can be a big help (and deliver cost savings).
In most cases, using an open-source solution only—a Selenium-based testing framework, for instance—is not going to help you achieve the test coverage you need in this situation. Selenium is for browser-based automation only, so if you need to test non-browser-based applications, it's not your best choice.
Many automation engineers need to automate a broad set of technologies, such as REST, .Net, Java, Oracle Forms, SAP, and mobile applications. Vendor-based tools let you easily combine all of them in one test.
When ease of use is paramount
Many open-source tools are really not much more than an API. Testing products using those tools requires having a team of engineers who are technically able to cobble together a custom framework that meets their needs. I’ve had a difficult time finding testers who have these types of skills.
Another benefit of using vendor-based tools is that good products simplify the test-creation process. Out of the box, they provide extra features such as built-in object spies, object-recognition improvements, reports, monitoring, and defect integration, without you having to do anything. These features enable you to have a total application lifecycle solution in place.
Also, many companies find that, because of the architecture of vendor-based tools, the business has an easier time finding qualified test engineers, since these products generally require fewer programming skills.
When you need certified-as-compliant testing platforms
Some companies require engineers to get approval for the testing tools they want to use. I know of one company that requires that all tools be submitted to an architecture review board before teams are allowed to use them.
In situations such as these, vendor-based tools may be a better fit, since they’ve usually gone through the rigors of meeting common compliance standards. In highly regulated industries such as healthcare and finance, choosing a tool that has proved to be safe, secure, and standards-compliant is an attractive option.
Most vendor-based testing tools have been around for a long time, and while some test engineers see that as a weakness, others see it as an advantage. These commercial tools have been battle-hardened and come with years of domain knowledge baked in. And most of the bugs have been shaken out of these products after years of development. That leads to the next reason to consider vendor-based tools.
When you absolutely can't put up with buggy tools
What happens when you find a bug in open-source testing software? You are at the mercy of the contributors. True, because it’s open source, you could theoretically fix it and commit the change yourself. But how many test engineers have this ability?
Vendor-based technical support is better able to handle bug fixes or quickly provide a workaround to customers. They have a paid technical staff that works around the clock and is solely dedicated to customer issues.
With a paid product, you can plead your case to the vendor, which will act with a sense of urgency to fix the issue— or at least add it to its product roadmap.
Try that with an open-source project and you might receive a snarky comment, or worse, encounter total indifference. It’s not that open-source contributors don’t care, but most of the time they're supported by volunteers who don’t get paid for time spent resolving your issues.
Open source is great, but many companies prefer to pay in order to have some measure of accountability. As the saying goes, sometimes you get what you pay for.
But don’t free tools save you money in the long run anyway?
When "free" software won't really save you money
Open-source tools are free, but do they really save you money? Not always. For example, consider cloud-based testing environment providers.
As more and more device types and operating systems arrive, with ever more complexity, the ability of your company to maintain its own test environments will consume more of your time, energy, and money.
Do you want to spend all your time maintaining test environments and patching and fixing issues before you even get to the actual testing? Cloud-based testing providers take care of all the maintenance so that you can concentrate on what matters to you most: testing and development.
The biggest benefit of going with vendor-based tools is the reduced cost of ownership associated with easier implementation and less administrative overhead.
Think about the business's needs first
It’s tempting to always use the latest and greatest tools. But before you proceed, be sure it’s the right decision for your team and company rather than just a way to build out your résumé or satisfy your curiosity.
Professional engineers should think of automation projects and test assets as part of the overall business. It's not about writing code or using the most fashionable technologies; it's about the fact that automation is a business solution.
How to make the right choice
Open-source tools are great. I’m a big proponent of using them to test software, and I use many open-source tools myself. But they're not always a total replacement for commercial testing tools. You just need to choose the right tools for your needs, whether those tools are open source or vendor-provided.
And you don't need to choose one or the other: Many vendor-based testing tools integrate with their open-source counterparts, which enables you to get the best of both worlds.
So don't dismiss commercial tools—or open-source tools—out of hand. Keep an open mind when evaluating your testing options, and you just might find that a mix of vendor-based and open-source tools is the best choice for your organization.
Agree? disagree? When do you reach for vendor-based testing tools versus open source? I look forward to your thoughts and comments.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
