Introduction
Every day, a vast amount of users open the App Store or the Google Play Store to leave ratings and reviews about the Mercari application. They generously provide insights into what they like and dislike, what they find valuable, and what they do not. At Mercari, our mission is to "circulate all forms of value to unleash the potential in all people."
User feedback in the form of app reviews is an invaluable connection between our customers and us. Incorporating this feedback into our development process allows us to unleash the full potential of our application and make it even more valuable to our users, which truly resonates with our mission.
Millions of users access our application daily, and chances are, you are one of them! Our users employ a wide range of devices, operating systems, and varying conditions when utilizing the Mercari mobile application (other open apps, available RAM memory, battery saving mode, phone configuration options, and many others).
Conducting comprehensive testing under all these circumstances is realistically impossible. While we can leverage the power of logs to understand user behavior and identify crashes, user feedback in the form of reviews provides a more profound insight. These reviews tell the stories of people who use and experience our application, sometimes sharing their positive experiences, and at times, encountering difficulties.
As QA Engineers and Managers, anything related to user value is of great interest to us. After all, quality can be defined as "Value to Some Person." In many cases, our users personify that definition. This is why, as a QA team, we have taken the initiative to integrate user feedback into our development process. Guided by the culture at Mercari, we have collaborated with various teams in order to build a process where everyone can harness the immense treasure trove that lies within Mercari’s application user reviews.
The end result is a process where we leverage ML to analyze, week by week, all reviews given to our app in the Google Play Store and App Store. The summarization and extraction of information improved cross functional communication between our teams, with the combined goals of addressing issues and considering improvements reported by our users, as well as constantly learning what is in the mind of our users and what they value of our products and services.
Previous Experience
One recent instance where user feedback took a protagonistic role on our software development process was on the run up to the launch of Mercari’s new application in 2022, code-named GU, short for "Ground Up". GU was more than just an update; it was a complete overhaul of the application. If you’re curious about the whole experience, be sure to check out our article below on the GU application development.
Related article (https://mercan.mercari.com/en/articles/35887/).
Recognizing the importance of involving our users in the development process, we knew that the success of the new app launch hinged on their collaboration. With a diverse range of devices, operating systems, configurations, and conditions, our users offered invaluable perspectives that we couldn’t simulate in-house.
To facilitate this communication, we launched volunteer-based beta programs on the App Store and Google Play Store, giving iOS and Android users the opportunity to try out the new application as soon as we had the new version of the app to a certain level of maturity. Users could share their feedback through a dedicated in-app feedback form.
What happened next exceeded our expectations. We were overwhelmed with the sheer volume of feedback from our users, presenting us with the challenge of sorting and categorizing the data into actionable information for our teams. Our software engineers were busy typing away at their keyboards, focusing on delivering our new version as fast as they could, making it impossible to invest in building an automated system to analyze user feedback. Instead, we had no choice but to tackle this task manually.
As a joint effort between our Voice of Customer (VoC) and QA Engineering teams, analyzing the feedback proved to be a complex task. We faced challenges such as identifying duplicate feedback, language barriers, effectively communicating the feedback to the right teams, and keeping up with the daily influx of feedback. It consumed a significant amount of our time. However, our efforts paid off. Not only did we uncover a fair share of hard to reproduce issues on the app, but we were also able to prioritize our tasks based on the quantity of reports received from users on specific topics.
It’s no exaggeration to say that the success of the app launch was partly attributed to the incredible Mercari users who generously shared their feedback. We were able to address problems, compatibility glitches with other apps, crashes, and even gather improvement suggestions. By taking their feedback into account, we avoided the chaos of encountering all of these issues on launch day.
Just imagine the turmoil that would have ensued if all users experienced these problems simultaneously! Thanks to the continuous iterative approach of listening to feedback and making necessary improvements, we seamlessly transitioned users into the new GU app. At a company level, we learned a valuable lesson about the immeasurable value of user feedback.
Hack Fest Project
After successfully launching the GU program and wrapping up Beta testing, I felt something was off with this user feedback channel shutting down. Though analyzing it was time-consuming, it provided valuable insight to our team. Instead of focusing less on this, I wondered if we could process more feedback with a more streamlined approach. To scale effectively, we needed to automate feedback categorization.
Initial slide of the presentation of our Hack Fest project: Feedback Classification (should have gone with a fancier name)
To gain traction on this notion, I pitched the idea on the ML Engineering Slack channel. Much to my delight, some colleagues showed keen interest, and one ML engineer named Paul Willot went further by becoming committed to making this a reality. Concurrently, I also discussed potential data sources with our Customer Support team and found support there as well, with Kazue Kudo san from their team deciding to champion things from the VoC side.
In the lead-up to our bi-annual Hack Fest event, a 3-day coding marathon open to all employees, we began shaping the framework of our project. With other new members, who jumped on board at the last minute, our objective was to build a minimum viable product (MVP) – an API that could autonomously categorize user feedback.
The initial step was to determine suitable categories for feedback. A balance between broad and overly detailed categories would need to be struck. We eventually settled on eight categories, after initially considering more than ten and doing fine tuning over them.
Next was the challenge of training our model. The estimate was that we would need approximately 100 reviews per category to showcase the model’s effectiveness. Combining our efforts, we managed to manually classify sufficient reviews across all categories, hopefully for the last time.
As the Hack Fest was drawing to a close, our ML engineer readied the model and prepared a proof of concept (PoC) site. Despite time constraints, we presented the working model and its potential benefits to our processes. Our project was well-received, even bagging a Special Mention from one of the judges (Maekawa Miho-san). However, we realized that this success marked merely the beginning of a greater journey towards fully implementing user feedback into our software development process.
If you are interested about Hack Fest and Mercari Culture, check out this related article: https://engineering.mercari.com/en/blog/entry/20230410-b286fe9577/
Template of the reports shared in Slack. This one breaks down the feedback in different categories for teams to work on. Monthly, we would share some graphs as well.
In the following months, our model filtered user feedback related to bugs or improvements, assisting in compiling weekly reports. This helped us to quickly evaluate weekly feedback by diving into the reviews that were classified as the categories we cared about the most. We picked out the topics most mentioned by our users and prioritized investigations on them. We also helped prioritize existing tickets with this information. Additionally, we collected improvements laid out by our users to enhance the capabilities of our apps. User input became an essential part of our development process, helping us detect issues and fine-tune features. While we acknowledged the potential to do more with this invaluable tool, we also identified some limitations and areas for future improvements.
Getting Real
Despite the positive response we received from our weekly reports, there were evident gaps in our process. But, in the spirit of innovation and continuous improvement, we recognized this as an opportunity to refine our approach.
One key issue was that stakeholders had limited access to the raw reviews. To address their queries, we had to manually sift through the reports, an exercise that was far from efficient. Similarly, recurrent albeit less common feedback often got overlooked due to its sparse occurrence in our weekly reports. Moreover, our process lacked instant, visually appealing data presentations like graphs, timelines or charts to help quickly comprehend the story behind numbers. Lastly, our not yet mature process had a tendency of duplicating data unnecessarily, a clear area for improvement.
Keen to tackle these pain points, we decided to revolutionize our weekly feedback reports. Our vision was to create an interactive dashboard that allowed stakeholders to access, filter, search, and analyze user feedback data as per their needs. We believed this would help unleash not just the potential of user data but also our team’s innovative thinking.
We began the transformation by defining the specifications for the new dashboard. To ensure its efficiency, we created mock data sets for testing and refinement. The upgraded dashboard would now enable users to filter reviews by dates, ratings, categorization, and relevance tags. Realizing the importance of seamless communication, we even integrated a feature for conducting keyword searches in both English and Japanese.
Keeping data duplication in check was our next goal. Following several productive discussions, we worked in cohesion with the VoC team and QA to co-own the user reviews data. This allowed us to modify the data without creating redundancies. We also added a "Feature" column as suggested by the VoC team, to clearly highlight which application functionality the feedback pertained to. As part of this journey, we automated some processes previously handled by the VoC team, in collaboration with QA. This teamwork was a testament to Mercari’s collaborative culture.
Sneak peek of the dashboard we have in place, showing the filtering options available and some basic graphs. Below there’s much more information, insights on ratings trend, preview of the reviews, etc.
With the refined data at our disposal, we integrated it with the dashboard, refining as necessary. We were thrilled to roll out our revised weekly reports complete with interactive dashboards. The overwhelming response from Product Development Teams, all roles included, was delightful to witness. Now, our teams could effectively filter and search for keywords related to new features, shedding light on invaluable insights. Empowering our team with this data seemed like a game-changing move and we’re already excited to witness its far-reaching impact.
Improvements for the Future
As we glance into the future, we see rooms for optimization and sophistication, even though we have significantly progressed from the days of manually classifying piles of user feedback. The quest for improvement never stops!
A call to augment our efforts in communication can’t be denied. Engaging as our internal product feedback channel might be, we also acknowledge that we haven’t roped everyone into it. The key lies in promoting our dashboards and making them more accessible, so that the wealth of insights within can benefit a wider audience. Moreover, a system to ensure that the highlights from our weekly report digests reach the teams who can act upon them directly, is in the works.
In terms of opportunities, our sights are set on pouring more data into our ML model pipeline. We strongly believe there is power in volume. By automating more, we can efficiently review additional feedback and take action instead of being bogged down by manual tasks. One promising proposal is to bring back the in-app feedback form. While another involves widening our range to cover feedback from other platforms like SNS. Indeed, our current model may need some fine-tuning for these new data sources, but the potential wealth of unique insights is an exciting prospect.
Finally, an alliance with the Customer Support (CS) team feels like a natural progression. CS creates investigations for repeated inquiries about the application behavior, and we do something quite similar for repeated user reviews of our app. We believe a collaboration may pave the way for a unified process or cross-utility of our data sets to boost the quality of the solutions we’re able to offer.
Conclusion
In summary, this journey of continuously refining our approach to harnessing user feedback stands as a testament to our unwavering commitment to quality at Mercari. The narrative underscores the profound impact of collaboration between different teams within the company, as well as the potential for continuous improvement in our delivery process.
From the initial struggle with overwhelming feedback, the manual categorization of data, to the development of an automated system through our Hack Fest project, we’ve come a long way in understanding and integrating user feedback. The creation of an interactive dashboard has positively revolutionized our weekly feedback reports, enabling the different teams to efficiently use and discern data. Furthermore, the role the VoC and QA teams played in refining our use of data underscores the richness of diverse collaborations within Mercari.
However, we recognize that our journey is far from over. There remain opportunities for more efficient communication, further data integration, and exploration of fruitful alliances with more teams within the company. As we continue evolving and embracing these opportunities, our aim remains clear: to deliver ever-greater value to the users of our application.
By sharing our unique experiences, we hope to spark dialogue and collaboration beyond Mercari’s walls. The strength of user feedback should never be underestimated. When harnessed effectively, it has the power to transform services, enhance user experience, and prompt targeted enhancements that truly resonate with user needs. As such, it is an invaluable resource in our mission to "circulate all forms of value to unleash the potential in all people”.
