Acceptance criteria: QA’s quality boost

Hello everyone! I’m @____rina____, a QA engineer at Mercari.

Welcome to article #1 in the series Behind the Development of Mercari Hallo: Flutter and Surrounding Technologies and day 3 of the Mercari Advent Calendar 2024!

Recently, on November 15, I gave a talk at Tokyo Test Fest, titled "Acceptance Criteria: QA’s Quality Boost." In this session, I talked about how important acceptance criteria is. QA writes this, and it helps in the whole development process, not just in Flutter. It’s also important for the whole team to review it together.

Acceptance criteria is important for teamwork in the development team. If we define it well and share it with everyone, we can really improve quality. In my talk, I also used real examples from our project to show how this process works.

I previously wrote an article about acceptance criteria; it’s only in Japanese, but if you’re interested in reading more, you can check it out here.

In this post, I’ll share a transcript of my talk.

Acceptance criteria: QA’s quality boost

Hello everyone at Tokyo Test Fest! I’m Rina. Thanks for coming today! Let’s get started with my presentation on the topic, "Acceptance criteria: QA’s quality boost."

Our QA Team’s initiative

Now, I would like to talk about one initiative that our QA Team is undertaking. More specifically, on how we document test cases in the acceptance criteria and have the entire development team review them together.

Acceptance criteria are often used in Scrum and Agile development. Before introducing the acceptance criteria, user stories and test cases were separated. This caused misunderstandings, especially during testing.

This new process helps the whole development team! Product managers, designers, engineers, and QAs—we all work better together. Let’s look at the benefits this process has for each team member.

For example, product managers find it easier to check specifications and continue development without missing anything. Previously, issues with the specifications were sometimes only noticed during the testing phase. This activity helps us find and fix mistakes early, so we don’t have to go back and redo our work.

Frontend and backend engineers are able to agree on the implementation plan beforehand, which makes the development go smoothly.

Additionally, by confirming specific wording and display methods on the spot, we can incorporate real-time feedback from designers, leading to higher-quality product development.

For QA engineers, sharing the ways to create test data and executing tests helps to improve our work during the testing phase. Previously, they had to consult developers about creating test data during test preparations. This new approach made it easier to talk about the order of development based on how easy it is to execute tests.

The whole team now understands complex projects better. This makes communication easier. When we have many projects at the same time, it’s easier to see how each project is going. This helps us work smoothly.
Now, let’s take a closer look at how we implement this initiative.

Three simple steps

We did three things.
First, we started including test cases in the acceptance criteria. Second, we changed how we do reviews. Before, we reviewed separately. But now, we review together. Finally, we made it a rule that the whole team participates in the reviews.
Where do you usually keep your test cases? Who uses them and how?

For example, maybe you use a test management tool. Or maybe you use a Google Spreadsheet or Excel file. Test cases are kept in many different places, and people use them in different ways. By sharing test cases with everyone, like with developers and product managers, they become more useful. They are helpful, and when everyone uses them, it’s even better. QA engineers know.

Previously, the connection between user stories and test cases was weak, increasing the risk of missing important tests and leading to having to redo our work later in the development process. Test cases were like a hidden treasure map. Despite being available to everyone, their value wasn’t used. Teams had trouble with their user stories (islands), not realizing the help they needed was right there.

Before, finding the right test was hard. It was like a treasure map with lots of islands. Each island had treasure, but it took a long time to see what was on each one. Now, we have a sign for each island! The sign is the acceptance criteria. It tells us exactly which tests we need for each user story. For example, the sign tells us what the product should do, what it should not do, and how to test it.

This makes it easier for everyone to understand and build a good quality product.

Example: Acceptance criteria

This slide shows an example of our acceptance criteria. We clearly define the test target, the condition for the test, and the expected result.

For example, in the first row, the test target is the display of the title and label. We expect both the title and the label to display "Login". In the second and third rows, we define the expected behavior of the screen based on the condition of the feature flag. Finally, we test it on iOS and Android to make sure it works the same way on both.

With clear signposts (acceptance criteria), our team navigates development more effectively. We’ve seen improved collaboration, fewer errors, less rework, and faster delivery of high-quality products. Everyone understands the goals and how to reach them—together.

Three simple steps to effective reviews

Now, I’ll talk about how we do reviews. Reviews are super important for our team. They help us all agree on what "good quality" means. This way, we can build a really good product. We review the acceptance criteria and test cases together. This helps us avoid mistakes and problems later. It also makes our work faster.

Let’s talk about how we do reviews. It’s really simple! There are three steps. First, we read it out loud. One person reads each acceptance criteria out loud. By doing this, you can see the acceptance criteria and test cases for each user story. The reader explains each item briefly.

Second, we ask questions. After reading, everyone can ask questions. Developers, QAs, product managers, designers—everyone! It’s good to have different viewpoints. For example, "What data will we use for this test?" or "Do we need this part?" Or even, "Will users understand this?" This approach helped us to have a good discussion as a team.

Third, check out. We review and confirm that everyone understands the acceptance criteria.
Then, the review is finished. These reviews help the team agree about quality from the beginning. That’s it! Three simple steps. Anyone can do it!

Let me explain why our new process is effective. We have a specification review to examine the initial requirements. But sometimes, engineers and QAs don’t understand all the details yet. It’s like looking at a picture that’s not clear.

After this, developers write design documents, and QAs write acceptance criteria and tests. In this way, everyone understands the features and user stories much better. Our new review happens after this. Everyone comes to the review with a clearer picture. Like a high-resolution picture! This makes our discussions better and more focused. We can find problems and improve the details together. Having the review later, when we all understand the details, helps us avoid rework. It improves quality and saves time!

But does this review process work for everyone, even without special skills?

We tried this review with team members of all skill levels. For example, I am a QA engineer, and I started doing this in my scrum team. At first, I did it by myself. But my whole team was able to see good results from it. Now, other QA engineers are doing it too. At first, some people were unsure. But now, everyone does the reviews smoothly. So, why does it work for everyone?

The key is understanding together. We write test cases with the acceptance criteria. This way, the whole team sees the same information. The whole team can discuss this at the same level. That’s why it works for all skill levels.
Of course, we can still improve the process. We want to make it even better! However, I believe this review method has great potential. It helps the whole team focus on quality and improves our development process.

So, here are the key takeaways. I talked about using acceptance criteria and test cases to improve quality. By putting test cases with acceptance criteria, everyone can easily see what to test. We write the test cases directly into the acceptance criteria. This helps the whole team build a better product. So, It’s easy to see and connect it to the user story.
While the whole team reviews together, we have good discussions. Everyone understands the product better. These changes help us agree on quality from the beginning. They help us avoid rework and save time. We want to make this process even better! Please give it a try in your teams and see if it improves your quality too.

Thank you for listening.

We hope this article has been helpful to your projects and technical explorations. We will continue to share our technical insights and experiences through this series, so stay tuned. Also, be sure to check out the other articles in the Mercari Advent Calendar 2024. We look forward to seeing you in the next article!