The 21st post in the Mercari Advent Calendar 2020 is brought to you by @funaki, @sawachan, @hkondo134, @yukanish, and @rina from the Mercari QAE Team.
For a long time, Mercari’s QA engineers were working on QA within projects. This meant mainly testing. However, Mercari adopted Scrum development starting in 2019, and testing is currently being conducted by Scrum team engineers and PMs.
After that change, the role of QA engineers is no longer just testing. We now focus on work to ensure the quality of the Mercari product and service as a whole.
VP @wakasa wrote a blog post about quality assurance. Please take a look if you are interested.
Quality Assurance is Engineering Excellence
Today, we’ll be introducing the activities that QA engineers are currently working on.
Scrum
QA engineers now work mainly outside of Scrum. Our activities have changed to focus on test plans and test case reviews.
Mercari divides the product into multiple areas, which we call Camps.
Test design and execution is carried out by the Scrum team engineers and PMs. QA engineers review functional requirements, specifications, and test cases. We also participate in scrum events such as refinement and sprint reviews to point out any concerns and give feedback from a user’s point of view.
QA Portal
The knowledge of QA engineers is now more accessible to everyone.
The QA engineering team has a lot of knowledge on how to run tests. We need to share that knowledge with the scrum teams. Therefore, we decided to use Confluence to create and update our QA portal site internally. Since there are many English speakers, all documents are written in English first.
- Onboarding: Information and links to get started with QA
- Various test accounts: Information for using test accounts
- Features and screens: A list of features and screens in Mercari
- Test Environment/Tools/Devices: How to collect tools for test execution and device management information
- How to test: Information and tips about test execution procedures for various functions of the product on the test environment and production environment, how to install the application, how to create tickets, test plan/test case examples, bug report examples, an explanation of the application release process, etc.
- JIRA Workflow: A description of the ticket development workflow
- Automation Testing (E2E Test): Information about E2E tests for iOS and Android
-
FAQ for QA: Frequently asked questions
In order to improve the QA portal site, we use Slack reactions to automatically collect topics to be added to the site, conduct questionnaires, and carry out educational activities.
Automation Testing(E2E Test)
Most QA engineers have started writing test code.
Regression test automation is created and maintained by QA engineers (though engineers also implement E2E tests). Previously, Automation engineers wrote the test code, but now it’s a QA engineer task. The members involved in this have different career backgrounds. There are Automation engineers, QA engineers who used to be developers, and QA engineers who started writing code for the first time.
iOS
iOS automation members write and run tests, identify run failures, and fix test code as needed. In the past, there was a time when running tests using CircleCI was not possible due to environmental problems. As a result, the person in charge had to execute the tests on their local environment, which, along with making fixes, took all day. Now that we’ve solved our environment issues, tests run with a nightly build job on CircleCI. With more members maintaining tests, we now have a high success rate.
Android
We write test code in Kotlin using Espresso. We previously wrote test code in Ruby using Appium, but we decided to migrate to Espresso because we wanted to place the tests in the same repository as the application, and also use the same language used in development, Kotlin, for UI tests. Also, the content and explanations of the scenarios were written in Japanese when using Appium, but since we migrated to Espresso, we write them in English to make it more accessible within the company. We hope that engineers will find it easier to work on UI/E2E testing by using the same repository, programming language, description language, and natural language.
We have now completed the migration of our test code to Espresso and are working on fixing the failed cases as well as refactoring the test code to increase stability. We also use Google’s Test Lab, a cloud-based infrastructure for testing apps, to run the tests, but there are still some challenges. Test failures could be due to timing issues, test lab device-dependent issues, or shared resource allocation issues. It is effective to apply parallel execution of tests run in a cloud-based environment; however, at the same time, these issues must be overcome in order to make full use of it.
Regression test management
We are still working on this in 2020. We translate test cases into English and review their validity to make sure they are up-to-date.
We use TestRail as a test management tool. We use the same test cases for regression tests on iOS and Android.
There are mainly two purposes for creating test cases: the first one is for automated tests that run on a daily basis; the second one is for release judgment tests, which are used as criteria for deciding if we will release updates to the app. Release judgment tests can be executed automatically or manually, and the appropriate test cases are identified and used as needed.
Client release facilitate
In 2020, business partners are continuing to take the lead in working on this.
The Mercari app currently has releases once every two weeks. Since one release includes multiple features for Mercari/Merpay, the release process must be organized. We’ve covered the release process on our blog: （アプリを安全にリリースするための取り組み(Release trainとClient release process)）
We call the process of managing new release versions "Client Release Facilitation". QA engineers (currently contracted QA engineers) are in charge of this. We also created a process for the tasks that client release facilitators are supposed to do. The Client Release Facilitation team looks back at each release and continues to improve this process.
Bug Analysis
In 2020, we are creating a process for bug analysis.
Mercari uses JIRA to manage tickets for tracking bugs. QA engineers are working on bug analysis so that we can propose quality improvements to the development team. This requires selecting methods and indicators for analyzing bugs, setting up environments for visualizing the status of bugs, and reconsidering bug ticket management rules.
We use a BI tool called Looker as a visualization tool. Since we have to combine various data sources for bug analysis, we require flexible customizability for visualization. Looker is a good tool for bug analysis because we can freely combine data to create graphs and manage them easily on a dashboard. Currently, we are investigating how to create useful graphs with Looker and how to utilize data from JIRA. In the future, once we collect data that can be analyzed for bugs, we will judge the quality status of developed features based on the amount of bugs occurring and make suggestions to the development team to improve the feature quality.
Device management
Scrum members now also have test devices, and we can use devices even when working from home.
At Mercari, there are currently about 300 testing devices for QA, and there are about 80 OS versions. Users have various devices and operating systems. We manage purchasing devices and updating OS versions to ensure behavior on as many devices and OSs as possible.
The devices are not only used by QA engineers, but also by engineers and PMs from each Scrum team. QA engineers check the location of the devices every two weeks. Although we have been working from home more this year, QA engineers are also selecting cloud verification device services so that testing can be executed efficiently.
Business Partner Management
QA engineers conduct one-on-one meetings to make it easier for business partners to work.
The QA engineering team works with multiple business partners (from an outsourcing company). We hold one-on-one meetings to make it easier for business partners to work with us. In these meetings, we ask our business partners what they are worried about, if they have anything they want to propose to the QA engineering team, and how they are doing. We have received good suggestions from our business partners, and they are working with the QA engineering team to improve the quality of our products. This also allows us to expand the range of our activities. Read more about how we work with business partners here:
Retrospective
As a result of retrospectives, team members have been more proactive about organizing team activities.
The QA engineering team conducts retrospectives once every two weeks. During retrospectives, we look back on the events during that period and come up with action items to improve. We use a tool called Retrium.
Retrium comes in several formats for retrospectives. Retrium has many useful functions such as anonymous voting, drafting, grouping, adding action items, timing, confirming action items, and so on. All participants can access these functions online simultaneously. We can easily create a KPT like the one shown in the figure.
As a result of continuing retrospectives, the QA engineering team has improved communication, and has created more opportunities to share knowledge and hold study sessions.
This image shows a study session on creating a release bot using Python. Some kind of study session is held every week.
Starting July 2020, there are now two people assigned to facilitate the retrospective; one facilitates the meeting and another operates Retrium. Then we look back on the retrospective afterwards and continue to improve the retrospective itself.
We often forget what has happened in the past two weeks. Therefore, we make a retrospective calendar to talk about the events that occurred during that period and anything that became a hot topic in Slack. This helps us finish the retrospective in one hour.
Conclusion
Previously, we were doing some of the tasks we described here while also doing QA for projects. Now we can focus on issues and improvements that we couldn’t tackle before. All of the activities introduced here are carried out by multiple team members, with some members involved in more than one. There are many new activities and challenges that we would like to work on in the future.
Mercari is looking for QA engineers who share our mission and values.
Tomorrow’s blog post —the 22th in the Mercari Advent Calendar 2020— will be written by ＠moricho. Hope you are looking forward to it!
