Introduction:

At Hyphen, we’ve undergone a transformative journey in testing methodologies, with developers leading the charge in end-to-end (E2E) testing. Initially reliant on a mix of E2E manual tests and automated tests in Java using Selenium and Cucumber, we encountered challenges with reliability as a result of manual testing, delivery speed, and comprehensive coverage. To address these issues, we empowered our developers to take charge of E2E testing, akin to how they handle unit and integration tests. Our choice of Kotest over Selenium marked a significant leap in test reliability and system coverage, aligning seamlessly with our Kotlin-based backend. Kotest supports the behavior-driven development (BDD) syntax (given, when, then) that Selenium has. Below is an example:

High level overview of Backend Architecture:

From an architectural standpoint, our backend system comprises a multitude of microservices built with Quarkus and Kotlin, interconnected predominantly through events emitted to Kafka. This event-driven architecture forms the backbone of our platform, facilitating seamless communication and data flow between various components.

To ensure a thorough validation of our E2E backend flow, we go beyond testing REST API endpoints and delve into validating the underlying systems. This encompasses validating the flow of Kafka messages and thoroughly assessing the behavior of our data sources, such as AWS MemoryDB (HA Redis) and PostgreSQL. By testing these components, we ensure the integrity and reliability of our entire backend infrastructure.

Challenges Faced:

• Ensuring the idempotence of our tests. It was crucial that running the tests multiple times (e.g., 100 times) yielded consistent behavior each time, without any flakiness. To achieve this, we seed test data for each test and delete the seeded data and any other data created in the test after the test stops running. Furthermore, we implemented a dynamic approach by spinning up a dedicated Kafka consumer group for each test. Once all tests are completed, we delete these consumer groups, ensuring a clean and isolated environment for each test case.
• Running these tests in the pipeline for all our AWS environments. We use Gitlab for our CI/CD pipeline and want to run these tests after a service is deployed. The main issue we ran into is allowing our Gitlab runner access to connect our AWS services such as Kafka, MemoryDB/Redis, Postgres, and Schema Registry.

Kotest and Kotlin:

As previously stated, we chose to go with Kotlin as the language and Kotest as the framework to write these tests. Kotest provides a powerful DSL for behavior-driven testing, making it easy to write expressive and readable tests. This allows product owners to write the tests in Confluence in English, and we can translate this to code. This is the primary reason why we chose Kotest as our test framework.

It is common in E2E tests to want to run actions before a test executes and/or after a test executes. Kotest supports the use of listeners (now called extensions) to perform actions before or after test execution.

Here are the listeners we used in our backend tests:

• Seeding data- To seed data before a test runs, we used a listener that ingests two JSON files per test (a member file to load member data, and a pharmacy file to load pharmacy data).
• Kafka listener- Since a lot of our communication between services is via Kafka, we often find ourselves needing to verify a specific event was emitted to one of our topics. To allow this verification to take place, we need to instantiate a Kafka consumer that subscribes to the topics we want to poll. This consumer spins down after the test is complete.
• Postgres listener- Since all these tests will have to interact with the Postgres database, we have added a Postgres listener that starts up and spins down for each test.
• Redis listener- Similar to the Postgres listener, since all these tests will have to interact with Redis, we added a Redis listener that starts up and spins down for each test.

Here is an example of one of the listeners: the Postgres listener. beforeSpec runs before the test as the name implies, and afterSpec runs after the test.

E2E Testing Strategy:

First, a product owner would write the test in given, when, then format (BDD) on a confluence page and provide the JSON test file for that test case. The developer would then write the test with the BDD specs outlined in the confluence page. Here’s a snippet of what a test would look like:

Integration with CI/CD Pipeline:

• For each AWS environment (even dev), we run these tests after a service is deployed in our CI/CD pipeline.
• After the test runs, we generate a pretty dashboard using a tool called Allure. It allows us to see the test results, and we also can feed it environment variables such as the environment the tests ran in, the user who executed the tests, and the repo that the tests ran in.

Lessons Learned and Best Practices:

• When creating E2E tests, you should treat the code with care, like you would for other code. Otherwise, maintaining them will be a nightmare.
• Put in the effort upfront. If we wanted to manually E2E test the whole. backend architecture, we could have (potentially) saved some time. However, in the long term, this saves so much time.
• Kotest and Kotlin is a powerful combination for writing E2E tests.