Mercari’s Adoption of Modern Testing Techniques

Introduction

Hello everyone! I’m @Udit, an Engineering Manager (QA) at Mercari.

In the ever-evolving landscape of software development, the role of software testing has become increasingly crucial. With the rapid adoption of agile and DevOps methodologies, the traditional approaches to testing have been challenged to keep up with the demands of today’s fast-paced development cycles. As a result, there has been a significant evolution in testing techniques, with a shift towards modern approaches that emphasize efficiency, scalability, and automation.

Mercari, a leading e-commerce platform, has adopted advanced testing techniques that have proven instrumental in enhancing the quality and reliability of its software products. In this blog post, we will delve into some of these advanced techniques, including advancements in API testing, frontend testing, dogfooding, release testing, and more. Through this exploration, we aim to highlight how Mercari’s innovative approach to testing has significantly contributed to the improvement of its software solutions’ quality.

Let’s dive deeper into some modern and cutting-edge testing techniques, one by one!

1. Shift-Left Testing

Shift-Left Testing involves moving testing activities earlier in the software development lifecycle, aiming to detect and address defects as soon as possible. This approach is crucial in Agile and DevOps methodologies for its benefits:

1. Reduced Costs: Early defect detection reduces expenses associated with fixing issues in later stages.
2. Higher Efficiency: Enables quicker identification and resolution of issues, leading to more efficient development cycles.
3. Higher Quality: Focuses on preventing defects early, resulting in higher overall software quality.
4. Competitive Advantage: Allows for faster and more reliable software delivery, giving a competitive edge in the market.

Examples include unit testing, integration testing, and code reviews, ensuring robust and high-quality software development. Additional practices such as dev/QA kickoff, developer self-check, and running sanity tests by developers further enhance our testing process.

2. Test Automation

Test automation is integral to modern software development, expediting testing processes and enabling faster feedback loops. This approach utilizes specialized tools and frameworks to automate test case execution, reducing reliance on manual intervention and enhancing testing efficiency.

At Mercari, we follow similar principles and practices in our test automation endeavors. This includes:

1. Consulting: Offering expert advice on developing tailored test automation strategies and implementing best practices.
2. Tool Selection: Aiding in selecting appropriate test automation tools and frameworks based on factors such as application type, technology stack, and team expertise. Examples include frameworks based on XCUITest, Playwright, and Jetpack Compose.
3. Automation Strategy and Planning: Developing comprehensive automation strategies aligned with business objectives and project requirements.
4. QE Framework and Platform: Implementing quality engineering frameworks and platforms to streamline test automation initiatives and ensure consistency across projects.
5. Test Case Development: Creating robust and maintainable test cases covering diverse functional and non-functional aspects of the application.
6. Execution and Maintenance: Establishing automated test execution pipelines and processes for continuous testing, alongside providing ongoing maintenance and optimization of test automation assets.

3. Continuous Testing

Continuous Testing is vital in our software development process at Mercari, seamlessly integrating into our CI/CD pipelines. It ensures quality across the software delivery process, automating test executions from code commit to deployment. The benefits are significant, including faster release cycles, reduced risk of defects in production, and increased confidence in code changes. By automating test execution, teams receive timely feedback on code quality, allowing them to address issues early in the development process. Here are some key aspects of Continuous Testing:

• Continuous Testing is seamlessly integrated into our CI/CD pipelines.
• It ensures quality at every stage of the software delivery process.
• Benefits include faster release cycles and reduced risk of defects in production.
• Continuous Testing fosters confidence in code changes and prevents costly downtime.
• By automating test execution, our teams gain timely feedback on code quality.

4. API Testing

API Testing, integral to Mercari’s software quality assurance, ensures the functionality, reliability, and performance of APIs, crucial for modern software applications. Through rigorous testing, teams validate APIs to function seamlessly, accurately handle diverse requests, and excel under varying conditions. Here’s an in-depth exploration of API Testing:

1. Importance: API testing is crucial for verifying that APIs meet their functional requirements, adhere to industry standards, and interact seamlessly with other components of the software ecosystem.

2. Tools and Techniques: Various tools and techniques are available for API testing, including Postman, REST Assured, and Swagger. These tools provide functionalities for creating, managing, and executing API tests efficiently.

3. Company Examples: Within our company, we commonly use Jest and Typescript-based frameworks for API testing, providing robust features and ensuring comprehensive test coverage. Additionally, Go-based frameworks are also utilized, further enhancing our testing capabilities.

4. Testing Scenarios: API testing encompasses a wide range of scenarios, including endpoint validation, data integrity checks, functional testing, and handling error responses. By simulating different types of requests and assessing API responses, teams can identify and address potential issues before they impact end-users.

API Testing is integral to Mercari’s software testing strategy, ensuring APIs function optimally and integrate seamlessly within the software ecosystem.

5. Frontend Testing

Frontend Testing, pivotal for Mercari’s user-centric approach, ensures a seamless and intuitive experience across various devices and platforms. By rigorously testing frontend components, teams identify and resolve issues related to usability and performance. Here’s a closer look at frontend testing:

1. Significance: Frontend testing plays a crucial role in validating the functionality and appearance of user interfaces, ensuring that they meet design specifications and user expectations. By conducting thorough frontend testing, teams can detect and address issues early in the development process, minimizing the risk of defects reaching production.

2. Frameworks and Tools: Several frontend testing frameworks and tools are available to facilitate the testing process. Examples include XCUITest for testing iOS applications, Playwright for cross-browser Web testing and automation, and Jetpack Compose UI for Android UI testing. These tools provide developers and QA engineers with the necessary capabilities to write, execute, and maintain frontend tests efficiently.

3. Testing Scenarios: Frontend testing encompasses a variety of scenarios, including UI validation, cross-browser testing, and end-to-end (E2E) testing. UI validation involves verifying that user interfaces render correctly and display the expected content and elements. Cross-browser testing ensures that web applications function consistently across different browsers and devices, while E2E testing involves testing the entire application workflow from start to finish, simulating real user interactions and scenarios.

By leveraging frontend testing frameworks and tools and embracing a holistic testing approach, teams at Mercari bolster the quality and dependability of their frontend components, culminating in a seamless and delightful user experience for customers.

6. Exploratory Testing

Exploratory Testing is a dynamic and creative approach to software testing embraced at Mercari, involving simultaneous test design and execution. It enables quality engineers to explore the application under test in real-time, uncovering defects and identifying usability issues that may elude scripted testing approaches. Here’s a closer look at exploratory testing:

1. Importance: Exploratory testing is essential for discovering hidden defects and usability issues that may not be covered by scripted test cases. Unlike traditional testing methods where test cases are predefined, exploratory testing encourages quality engineers to think outside the box, follow their intuition, and explore the application organically. This approach often leads to the discovery of critical defects and provides valuable insights into the user experience.

2. Complement to Scripted Testing: While scripted testing provides structure and repeatability, exploratory testing complements it by allowing quality engineers to investigate areas of the application that may not have been considered during test case design. By combining both approaches, teams can achieve comprehensive test coverage and uncover a wider range of issues.

3. Effective Practices: Effective exploratory testing requires careful planning and execution. Test charters, which outline the areas of the application to be explored, can help focus testing efforts and ensure thorough coverage. Timeboxing, or setting a specific time limit for testing sessions, helps prevent quality engineers from getting bogged down in details and encourages rapid exploration. Additionally, bug advocacy, where testers advocate for the importance of discovered defects, helps ensure that critical issues are addressed promptly.

By integrating exploratory testing into their testing strategies, teams at Mercari enhance the quality of their software, boost user satisfaction, and mitigate the risk of releasing defective products. This approach nurtures creativity and critical thinking among quality engineers, resulting in more robust and reliable applications.

7. Dogfooding

Dogfooding, also known as eating your own dog food or self-hosting, is a testing technique practiced at Mercari where developers and other stakeholders use their own software in real-world scenarios. Here’s a deeper dive into dogfooding testing:

1. Introduction: Dogfooding involves using the software products that you develop within your organization. Instead of relying solely on traditional testing methods, such as automated and manual testing, dogfooding encourages developers, quality engineers, and other team members to become end-users of their own products. This approach allows them to experience the software firsthand, identify usability issues, and gain valuable insights into its performance and functionality in real-world environments.

2. Benefits: Dogfooding offers several benefits to organizations, including the opportunity to gather immediate feedback from internal users, identify usability issues early in the development process, and validate the software’s functionality in real-world scenarios. By using their own products, teams can better understand the user experience, anticipate user needs, and make informed decisions about product improvements and enhancements. Additionally, dogfooding fosters a culture of continuous improvement and encourages collaboration and communication across different teams within the organization.

3. Examples and Best Practices: Many successful product companies have embraced dogfooding as a core testing strategy, where employees test pre-release versions of software internally to identify bugs and provide feedback before releasing them to the public. To implement dogfooding effectively, organizations should establish clear guidelines and procedures for using their own products, provide training and support to users, and prioritize feedback collection and analysis.

By incorporating dogfooding into our testing processes at Mercari, we enhance the quality of our software, increase user satisfaction, and accelerate innovation. This testing technique enables us to gain valuable insights into our products, identify issues early, and deliver better experiences to our customers.

8. Release Testing

Release testing plays a crucial role in ensuring the stability and reliability of software releases, especially in scenarios where multiple teams or companies contribute to a single application. This situation presents the challenge of coordinating changes from different entities, such as Mercari, Merpay, and Mercoin, all making modifications to our app and releasing simultaneously. Here’s a deeper exploration of release testing:

1. Introduction: Release testing is a critical phase in the software development lifecycle where the focus shifts from individual feature testing to validating the entire software product in preparation for deployment. Its primary goal is to ensure that the software meets the quality standards and functional requirements before it is released to end-users.

2. Release Testing Strategies: Release testing encompasses various strategies to verify the functionality, performance, and usability of the software. These strategies include:

   • Smoke Testing: A preliminary round of testing aimed at quickly identifying major issues or showstoppers in the software build. It verifies that the basic functionalities of the application are working as expected.
   • Critical Business Use Cases or Must Pass Scenarios: Verification of essential business workflows or must-pass scenarios that are crucial for the software’s core functionality and user experience.
   • End-to-End (E2E) Testing: Inclusion of end-to-end testing scenarios that simulate real-world user interactions across multiple components or systems to validate the software’s behavior under various conditions.
   • Regression Testing: A comprehensive testing approach that validates the existing functionality of the software after making changes or enhancements. It ensures that new updates do not adversely affect the existing features.

3. Automation: Automating release testing processes is essential for streamlining deployments and minimizing downtime. By automating repetitive tasks such as test execution, regression testing, and environment setup, organizations can accelerate the release cycle and improve overall efficiency. Automation also helps increase test coverage, reduce manual errors, and enable continuous integration and continuous delivery (CI/CD) pipelines.

By integrating robust release testing strategies and utilizing automation tools, we ensure the quality and reliability of its software releases. This proactive approach mitigates the risk of post-release issues and enhances customer satisfaction and trust in the product.

9. Production Testing

Production testing ensures software stability and performance in real-world environments, including sanity checks and basic performance evaluations.

1. Introduction: Production testing, also known as post-deployment testing, involves validating the functionality, performance, and reliability of software applications in a live production environment. Unlike pre-production testing, which occurs in testing or staging environments, production testing focuses on ensuring that the software performs as expected in real-world conditions.

2. Sanity Tests on Production: Sanity tests, also known as smoke tests, are conducted on the production environment to quickly verify essential functionalities and confirm that the system is operational after deployment. These tests typically cover critical use cases and key features to ensure that the basic functionality of the application is intact. Examples include user authentication, data retrieval, and basic navigation flows.

3. Basic Performance Checks: Production testing also includes basic performance checks to assess the responsiveness and stability of the application under typical user loads. These checks may involve monitoring key performance indicators (KPIs) such as response times, throughput, and error rates to identify any performance bottlenecks or degradation in system performance. While more comprehensive performance testing may occur earlier in the testing process, basic performance checks on production help ensure that the application meets acceptable performance standards in the live environment.

4. Importance: Production testing is crucial for detecting issues that may only manifest in a live production environment, such as configuration errors, compatibility issues, or unexpected interactions with other systems. By conducting thorough testing in the production environment, organizations can identify and resolve issues promptly, minimize downtime, and maintain a positive user experience.

5. Continuous Improvement: Production testing is not a one-time event but an ongoing process that continues throughout the software lifecycle. By continuously monitoring and testing the production environment, organizations can identify areas for improvement, implement enhancements, and deliver a reliable and high-quality user experience.

Implementing robust production testing practices at Mercari ensures that our software applications perform optimally in real-world scenarios, enhancing user satisfaction and maintaining business continuity.

10. Post-Release Support

Following the deployment of a software release, the focus shifts to post-release testing support, a critical phase aimed at ensuring the ongoing functionality, stability, and performance of the software. Here’s a more detailed breakdown of the key responsibilities involved:

1. Continuous Monitoring: Implementing robust monitoring solutions to track system performance, detect anomalies, and identify potential issues in real-time, including crash management activities to promptly address any unforeseen incidents.

2. Customer Inquiries Handling: Promptly addressing and resolving customer inquiries and concerns regarding the newly released software. This includes providing timely responses, offering solutions or workarounds, and ensuring customer satisfaction, along with collecting Voice of Customer (VoC) feedback to gather insights for future improvements.

3. Issue Handling and Verifications: Actively managing reported issues by investigating root causes, implementing fixes, and verifying their effectiveness. This involves collaboration with development and operations teams to prioritize and address issues efficiently.

4. Support Hotfixes Deployment: Developing and deploying hotfixes as needed to address critical issues or vulnerabilities identified post-release. This may involve expedited testing and release cycles to minimize disruption to users and maintain the integrity of the software.

5. Performance Monitoring and Optimization: Conducting ongoing performance testing and optimization efforts to ensure that the software continues to meet performance requirements and user expectations. This includes identifying performance bottlenecks, optimizing code, and scaling resources as needed to maintain optimal performance levels.

By proactively addressing post-release testing support activities at Mercari, we effectively mitigate risks, maintain user satisfaction, and ensure the long-term success of our software products in production environments.

Conclusion

In conclusion, Mercari’s adoption of modern software testing techniques encompasses a wide array of methodologies and practices aimed at enhancing the quality, reliability, and performance of its software products. From shift-left testing to continuous integration and deployment, API and frontend testing to exploratory and production testing, each approach plays a crucial role in ensuring that software meets the evolving needs and expectations of users. By embracing these techniques, Mercari accelerates delivery cycles, reduces costs, and delivers superior user experiences, ultimately driving business success in today’s competitive landscape.