End-to-end vs. contract-based testing: How to choose

For the past few years a debate has raged over whether contract-based testing can replace functional, or end-to-end, testing. Some suggest substituting end-user-focused testing with unit tests or unit-test frameworks, written and executed during code deployments.

Consumer-driven contract-based testing is not new, but it has enjoyed a resurgence in popularity as microservice-based connectivity and API connectivity have increased. Think about it: Almost everything is connected, from your cellphone to web pages to home electronics. The intricate web of API and data connections continues to become more and more complex. 

Is it possible to replace end-to-end functional system testing with a suite of unit tests routinely checking integration points? Maybe. But before you decide to replace functional end-to-end testing using contract-based testing, you'll need to understand where they are similar and where they're not even close to the same thing.

Here is a comparison and of these two types of testing.

World Quality Report 2018-19: The State of QA and Testing

End-to-end testing

End-to-end testing involves testing a user workflow from beginning to end, including all back-end processing engines, be they API services or other messaging or data transfer services. This involves testing an entire application from the beginning of the user workflow process all the way to the end.  

It's a full system test, typically executed with a combination of unit, automated, and manual test execution.  

Contract-based testing

Consumer-based contract-based testing is exactly what it claims to be: You test contract agreements between a consumer endpoint and an API provider endpoint. Contract-based test suites are combined to cover each interaction scenario that takes place between the consumer and provider endpoints. You're creating unit tests or suites of unit tests that validate that your API endpoint connections are functioning properly at any given time and according to your "contract."  

Interfaces that cannot be used directly because they are "production" endpoints can be mocked or faked and still tested.  

Contract-based test execution tells you that the API endpoint code:

  • Works as expected.
  • Includes endpoints that are up and active.
  • May have broken a previously working connection on either the consumer or provider end because of changes made.

Contract-based tests typically won't detect endpoint configuration issues or misused classes. Other types of testing cover those errors and catch those types of defects. Consider how your product's users may benefit from combining both contract-based and end-to-end tests. 

One can't substitute for the other

Contract-based unit tests only check that API endpoint connections are active and functioning correctly. But that's not all the testing your software application requires. It's highly unlikely that contract-based unit tests have checked any customer workflows or customer-facing functionality. However, there are ways to combine the testing types you need to ensure the quality of your application release. For the best testing coverage, consolidate.

Can you replace automated API test executions? Possibly. If the provider changes its code and breaks your application, contract-based tests can find those issues. But contract-based unit tests may not be able to tell you where the break is. The automated API test may, however, include information to help you locate the breaking point.  

So, before you replace API-based functional tests with contract-based tests, ensure that the tests they execute are duplicates of each other. Consider keeping tests that are not duplicated, so they catch issues that might not be covered by the others.

Contract-based: How much coverage do you need?

Contract-based tests also require contact between provider and consumer for test execution. Either that or they must be tested against a mocked or faked system and not against the production instance of an external service. If you need to hit the production instance, contact the provider before hitting its endpoint with tests. You’ll both need to be aware of the impact of your tests and what, if any, data is added or changed.  

Another consideration with contract-based testing, as with nearly any other type of testing, is depth. How much test coverage do you need? It’s important to test your side of the API, but how far into the full functionality of both endpoints do you test?

One of the arguments against contract-based testing is that it doesn’t cover the full functionality and only scratches the surface. Following full functional paths tends to lead the test developer down potential rabbit holes, resulting in tests that break frequently and are hard to maintain. Design your contract-based test approach to the depth desired and note where to supplement testing for optimal test coverage.

Use contract-based tests to focus on catching defects:

  • Within the consumer workflows
  • Around misunderstandings regarding endpoint configuration or payload
  • When the provider has created breaking changes on endpoints and/or payload

Get the full workflow

End-to-end functional testing covers more than API functionality, or other microservice-type exchanges. End-to-end testing covers the full end-user workflow from beginning to end, regardless of which back-end systems you're crossing or using. The focus of an end-to-end testing effort is on the end-user workflow functionality and not the particular interface doing the processing behind it.

Frequently, you'll find defects that are not necessarily logical and are present in the UI workflow. Unit tests cannot find non-logical defects, nor can they detect problems with database communications.  

For example, if you execute end-to-end tests, you'll know quickly—when tests continue to fail—that the database is not communicating. End-to-end tests are designed to be thorough and complex, which makes them time-intensive. However, they tend to find significant defects that otherwise go undetected.

Contract-based testing, on the other hand, is less intensive and can be done more frequently, providing extensive smoke testing for coverage each time code is deployed.

Use both with minimal overlap

Your best overall bet is to determine what functional API or other microservices testing can be effectively tested with contract-based unit tests so you’re not duplicating testing efforts. For example, end-to-end functional testing will uncover misconfigurations that appear in resulting data or the user interface. Code reviews are useful for uncovering nonstandard class and object development. 

Contract-based testing cannot replace functional end-to-end testing, but you can combine the two to improve testing coverage across the board.  

Topics: Dev & Test