You are here

You are here

Magic Quadrant for Test Automation: How Gartner got it wrong

public://pictures/Joe-Colantonio .jpg
Joe Colantonio Founder, TestGuild

In my last TechBeacon article, I wrote an in-depth review of the Gartner Magic Quadrant for Software Test Automation Report, in which I dissected the full report. But one thing I didn’t mention was that Gartner was actually wrong — with a few small caveats. 

The time for open source tools is now

The Strategic Planning Assumption section of the Gartner report states that by the year 2020, Selenium WebDriver will become the standard for functional test execution and that this will marginalize vendors that can't provide strong, higher-level test functionality. I believe Gartner's date is incorrect; in fact, some might say that Selenium is already the de facto standard for web-based test automation. It would follow, then, that the time for vendors to embrace open-source tools like Selenium is not 2020 — it’s now. 

Another reason why I feel the 2020 date is way off is that, as I mentioned in the original article, there is currently a W3C WebDriver standard in draft form; once this is approved and becomes official, standard browser vendors will be responsible for creating their own browser-based implementation of Selenium WebDriver. I believe the WebDriver W3C working draft will become official way before 2020 and once it does, vendors will be better able to support it.

Once WebDriver becomes standard, I’m hoping it will make automating the browser much easier (even for Internet Explorer), because each browser vendor will own its respective implementations and have a vested interest in them.

If Microsoft can embrace open source, so should you

In fact, Microsoft has already announced (in July 2015) that it is supporting the Microsoft Edge browser through the W3C WebDriver standard. It has also explained its rationale, saying that it did it because it’s a best practice and because all web content, browsers, and specifications should align to the same well-defined behavior. This is incredible — and a significant win for open-source proponents.

If you’re old enough, you may remember how militant Microsoft’s stance was against all things open source-related, but this move — along with what I’ve actually heard from several Microsoft employees — is an excellent example of a complete culture change to the embracing of open-source technologies. Open source is the future, and that future is now.

In a further show of support of the growing trend in open-source containerization, Microsoft will be adding support for Docker in Windows Server 2016. Testing tools and technology have always seemed to lag behind developer technology, but test tool vendors need to adapt to this changing open-source world as Microsoft has if they want to survive and compete in the coming years.

Why no Selenium, Gartner?

I also feel it was a mistake for Gartner to leave Selenium out of its report as a viable option for software test automation. I understand that it didn’t meet the criteria Gartner clearly listed, but to my (and many others') way of thinking, it was a glaring omission.

I also think it’s clear to anyone who has been involved in automation for any length of time that many companies — even large ones — are moving closer to utilizing open-source solutions in their development efforts. To leave Selenium and other open-source options out is, as far as I’m concerned, backwards thinking. It’s definitely not in line with what I’ve been seeing in the testing field over the last few years. 

Combine technologies to create a full solution

There are also many ways in which you can combine multiple open-source projects to create an automated testing framework that could meet Gartner’s key use cases for tools needing to support mobile applications, responsive design, and packaged applications like ERP and SAP.

For example, many SAP applications are web-based, and the majority of those applications could be automated using Selenium Webdriver. If you were to encounter controls that are not supported or recognized by Selenium, you could always use another open-source, image-recognition-based tool such as SikuliX. SikuliX is powered by OpenCV, can automate anything that can be seen on the screen, and works on Windows, Mac, and Linux.

For mobile, there are a ton of open-source options for testing both Android and iOS apps, such as Appium, Selendroid, Robotium, and Frank, to name just a few.

Test management and requirements can be achieved using Cucumber, jBehave, and Specflow, which are all open-source, behavior-driven developer frameworks. There is also an open-source solution called the Galen Framework that was created with responsive design in mind.

Open source is a reality, but it’s not as free and easy as some folks would make it seem. While cobbling together a bunch of open-source solutions to create a framework that can meet many companies’ automated testing needs is a challenge, it can be done.

This is where I feel that vendors can step in to fill the void; their support of open-source technology would make it easier for people to get their automation efforts up and running.

Proprietary is the past

After publishing my original post, I also heard from a few people who mentioned that this year’s Gartner report was too focused on automation from a software development perspective as opposed to a business perspective.

They pointed to advances in “code-less” automation as being the future. Sorry, but I don’t see it. While it might have its place in simple test workflows, test automation is part of development and needs to be treated just like any other development efforts — using the same rigor that would be expended on production code.

Furthermore, any solution — even a code-less one — should be compatible with a company’s open-source development ecosystem. 

Why open source's day has come

Many companies have already begun embracing open-source transformation, and I believe that’s critical, because the non-inclusive vendor doomsday clock is rapidly approaching the point of no return. I feel that for next year’s Test Automation Magic Quadrant to be seen as credible and relevant, Gartner should begin taking open-source technologies more seriously in its analysis and include them as viable options for test automation.

Image credit: Flickr

Keep learning

Read more articles about: App Dev & TestingTesting