6 top open-source testing automation frameworks: How to choose
If you're thinking about building your own test automation framework, think again. You'll do very well in most cases to consider one or more of the open-source options available. That's because, in general, a framework is a set of best practices, assumptions, common tools, and libraries that can be used across teams. In most cases you don't need one that's unique to your dev environment.
You simply need a framework is to help make your test automation code:
I’ve seen some teams take these benefits to the extreme by building their own, elaborate automation frameworks from scratch. This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code—and in most cases with better results.
Before you fall into the “build your own framework” trap, be sure to check out these six open-source test automation solutions.
If you need a Java-based framework that integrates with behavior-driven development tools such as Cucumber and JBehave and that lets you keep your test scenarios at a high level while accommodating lower-level implementation details in your reports, Serenity might be the tool for you. Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.
A key benefit is that Serenity acts as a wrapper on top of Selenium WebDriver and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started. Serenity also offers a lot of built-in functionality such as providing WebDriver management, managing state between steps, taking screenshots, running tests in parallel, facilitating Jira integration, and more—all without having to write a single line of code.
What Serenity is really awesome at is creating unbelievable reports. Out of the box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results, but also as documentation for your application.
It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.
Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open-source frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.
If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework.
The Robot Framework is a mature solution that was created for testers; it uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.
The Selenium WebDriver library is probably the most used external test library, but Robot Framework can test things other than websites, such as FTP, MongoDB, Android, Appium, and more. In addition to all this open-source awesomeness, it has a lot of APIs to help make it as extensible as possible.
The Robot Framework is Python-based, but you can also use Jython (Java) or IronPython (.NET).
The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.
If your team is made up of mostly testers, Robot Framework would be a great option for your automation framework.
RedwoodHQ takes a slightly different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.
Tests can be written in Java/Groovy, Python, and C# to test web applications with Selenium, APIs, or databases using their web IDE, which also includes built-in GIT integration.
You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script, you just need to find the action you want to perform, drag it into your test case, and enter the parameter values it expects.
The built-in IDE also allows you to create and modify test cases and actions as well as to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.
RedwoodHQ is tester-friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.
Developed for testing web applications, Sahi offers both an open-source and a pro version. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.
The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.
You can also use the controller to play back your script. Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications. But beware of relying on record and playback; this approach has proved to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.
I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, lots of folks rave about the Sahi pro version.
If your automation efforts are focused on user experience design (UX) or layout testing, Galen Framework might be a perfect fit for your needs.
Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed HTML reports with screenshots, and it includes a visual image comparison with a cool heat map feature.
Gauge is one of the newest open-source test automation tools for Mac, Windows, and Linux, and it was developed by the folks at ThoughtWorks—the same company that created Selenium.
Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living and executable documentation.
Gauge automation tests, called "specs," are written using a simple markdown language, using Java, Ruby, and C# within your existing developer IDEs such as Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.
Gauge is pretty new and is still in beta, so if your team is into cutting-edge test development, you might want to give it a try.
The preceding solutions are just a small sample of the types of open-source test automation frameworks available.
Before you write that first line of code in creating your own framework, make sure there isn’t a library or framework you can leverage. I suggest you stop wasting your time reinventing the wheel and check out one of these automation frameworks first.
Image credit: Flickr