* This article is a translation of the Japanese article written on December 24, 2021.
Hello! This is miisan, and I’m an Engineering Manager for the Merpay QA Team. For day 24 of Merpay Advent Calendar 2021, I’ll be discussing the current state of automation in the Merpay QA Team.
It’s been almost a year since I wrote “Why is Merpay QA involved in DevOps?” for Merpay Advent Calendar 2020. One thing the Merpay QA Team has been focusing on this past year is promoting test automation. In this entry, I will cover some of the issues that we worked on while developing test automation, and introduce some of the tools we use to support products.
Supporting Merpay with QA and automation
The Merpay QA Team is working on test automation for three reasons: reducing workload, accelerating release, and stabilizing quality. These are pretty straightforward, and they’re probably the reasons for automation at most companies.
Merpay is proactively involved in DevOps in order to “continue to provide faster, more reliable, and better services to customers.” To do that, we need to accelerate QA, streamline operations, and handle more tests with fewer people. This means that it is crucial to effectively automate tests in high-impact areas.
One major characteristic of Merpay QA is that we try to help get products out quickly (as opposed to QA that stops releases). We aim to create a system of cooperation with the Development Team so that there is mutual support for our responsibilities. We are also working to enhance quality by incorporating many tests during the development phase, and optimize testing during the verification phase by trying to incorporate automated testing from early on in development.
People tend to think that QA testing occurs during the final phase prior to release. However, the product development process is in a constant state of flux, and this results in a cycle of intermittent release. That’s why we need to think of QA not as “testing at the end,” but as “testing continuously.” If we rely only on manual testing in this kind of environment, we would need to devote a lot of resources to continue to keep testing. An enormous amount of QA resources would be required each time a product is scaled up.
Also, in order to automate, we need to establish the type of environment required to perform automation. Anything that was being handled through communication would need to be restructured to make automation easier. In order to make this possible, we at Merpay take advantage of test automation tools that fit well with our development culture.
Automation tools used at Merpay
The main test automation tools and project management tools that we use can be broadly categorized as backend, frontend, and native applications, as shown below.
I don’t think the tools we use differ greatly from the ones used at other companies. Having said that, one point worth mentioning that I think is very Merpay-esque is that we use “Scenarigo,” which was developed internally by one of our developers.
Scenarigo
“Scenarigo” is a piece of scenario-based test automation software developed as open source software in the Go language (Golang) by @zoncoen, a Merpay developer.
Scenarigo is a tool for running API server scenario tests. It is similar to Postman, which is often used for backend testing. Scenarigo has the following features.
[Scenarigo features]
• Test scenarios can be written in YAML
• Test scenarios written in YAML can be reused
• Can be expanded using Go instead of JavaScript
• Both HTTP and gRPC can be used
Test scenarios can be written in YAML
When managing test scenarios based on pull requests, Postman requires that you handle exported json test definitions. These json files have a complicated structure and are not developer-friendly, as it can be difficult to check for differences on GitHub based on pull requests.
In comparison, Scenarigo test scenarios have a more simple structure that uses YAML, which is generally thought to be more easily readable than json.
Test scenarios can be reused
One very convenient feature is that general test scenarios and common test processes can be reused.
For example, general test scenarios (such as creating users and logging in) and processes often used at the start of testing (such as granting the remaining Merpay balance) can be proactively reused to create efficient test scenarios.
Can be expanded using Go instead of JavaScript
Scenarigo is a very familiar tool for Merpay development engineers, who use Golang for actual development. It’s also possible to create Golang plugins, and call plugins created in YAML for test scenarios.
For example, a test scenario that would normally be difficult with Postman (such as testing the results of running a job after calling an API) can be easily implemented with Scenarigo by calling a plugin written in Golang. Thanks to this, we’ve been able to increase the types of test scenarios that can be implemented.
Establishing a tool environment that makes it easier for development engineers to maintain tests has also helped eliminate the boundary between development engineers and QA.
Both HTTP and gRPC can be used
For backend regression testing, we generally send a request to each microservice and verify the response result.
One unique characteristic of the Merpay architecture is that most microservices are implemented in Go and communicate using gRPC, and so being able to use both HTTP and gRPC is definitely a benefit (because HTTP requests need to be converted into gRPC by the API Gateway).
If you’d like to learn more about Scenarigo, please see the following link.
In-house tool for specific microservices
We also have a team using an in-house tool to realize automation for specific microservices. If you’d like to learn more about this in-house tool, please see the following link.
“Cypress + TestRail”
“Cypress” is an E2E testing framework that has become the standard for frontend E2E test automation.
“Cypress” is also used for testing during frontend development, so we can always ask for help or have our work reviewed. It’s very convenient.
We have divided up our responsibilities, with QA engineers in charge of regression testing prior to release, and product development engineers in charge of unit testing and integration testing.
We also use “TestRail” to systemize test case management in order to continue to improve workflows.
The following articles provide more details on “Cypress + TestRail” and the “Merpay frontend test automation policy.”
Issues for automation
Honestly speaking, we’ve run into a lot of issues while working on automation.
Maintainability
- This issue involves maintenance for test cases that have already been created. It’s very important to prepare an environment that allows for maintenance to also be performed by others. This is one of the major issues we’re facing now.
Individualization
- Most of our tests target microservices, so when something goes wrong with a test, it can be difficult to figure out where exactly the problem occurred. It can take time to identify the cause of a problem. And, even if we figure out the cause, it can take time to solve the problem if it occurred with another team’s work and you are forced to solve it without sufficient expertise. In that respect, we must work on reducing the cases of this happening, making things easier to understand, and eliminating individualized situations. We’ve also started to get an idea of the issues with using Scenarigo to create test scenarios.
High threshold
- One benefit of using Scenarigo is that it’s an in-house tool, so we can expect that it will be improved. Thanks to the “all for one” cooperation from our engineers, we can do more with the tool and it’s easier to use than two years ago. However, an unavoidable disadvantage of an in-house tool is that people outside the organization do not know how to use it. We’ve made up for this by creating manuals on how to use Scenarigo and holding study sessions and lectures.
Insufficient readability/visualization
- Another issue we’re tackling is how to handle the results of running automated tests. We use the results to make decisions before release, but we must also use automated tests to always keep the health of our services in check. It would be great if these test results could be viewed anywhere, at any time, and by anyone. I hope to build that kind of environment.
Future challenges
Although we’ve introduced these kinds of tools to automate testing, we still have cases where we write automated scenario tests based on manual tests. This is why I’ve asked our team members to try to incorporate a foundation for automation earlier on in the process, so that we can write automated scenarios in processes that are higher upstream. In the short term, using manual tests for all verification processes might allow us to release products and services more quickly. However, I hope everyone on the team will understand that automating repeated scenario tests will make future testing more efficient and help to ensure reliable quality. I want to focus on building an environment that allows for automated testing. We will probably continue to develop products at Merpay with speedy release cycles, so I think it will be necessary for each of us to think about our verification plans from a medium- to long-term perspective.
I’d also like to find a way to introduce automation throughout the entire product, instead of having to split automation between backend, frontend, and native applications. The main automated tests of the QA Team are still only used for the backend. I’d like to take a look at frontend and native applications so that we can work on automation methods that are ideal for the whole rather than just parts.
I didn’t touch on native applications this time, but these would be Espresso (Android) and XCUITest (iOS), which are used to write automated tests. These haven’t been introduced yet for QA, and I’m hoping to work with the Client Team so that we can write more and more test code.
Conclusion
So that’s the current state of automation in the Merpay QA Team!
This year, the QA Team worked as one to spread information in a variety of ways. For example, events such as “Merpay Tech Talk” and “QA Talk”—we provided a range of information on topics including our automation efforts, QA policies for individual products, and the concept of “All for Quality” that the QA Team works under.
I mainly talked about QA automation in this entry, but I also want to mention that we were able to build a strong relationship of trust between the Development and QA Teams and work closely together because of the Merpay culture of “All for Quality.”
“All for Quality” means that all team members involved in product development and operation work together to establish quality in order to provide customers with even better services, regardless of position or role.
You can learn more about “All for Quality” in the following links.
What do Merpay QA engineers talk about when discussing quality assurance?
I’d like to end by mentioning that Merpay is looking for QA engineers who agree with our mission and values. We’d love for you to join our team. If you’re interested, please contact us!
