The Mercari Group offers various applications and services, all of which are based on the Group’s payment infrastructure. This infrastructure is provided by Merpay. We interviewed backend engineers at Merpay, responsible for the company’s QR code payment and other payment services, and asked them about their experiences and creative solutions to issues that have arisen during the development process.
The interview was conducted by @afroscript of the Engineering Office .
Code Payment Team
— Could you first introduce yourselves?
@upamune: I go by the user name upamune. I joined Merpay immediately after graduating university in April 2018. Within the Code Payment Team, I am a backend tech lead and am responsible for ensuring that the development process progresses smoothly while maintaining communication with product managers (PM) and other teams. During university, I interned at various companies and worked fulltime at a blockchain-related company.
@ktr: I go by the user name kitaro, ktr. I joined Merpay immediately after graduating university in April 2019. As a backend engineer at Merpay, I am responsible for a wide range of activities, including design, development, and operation. Recently, I have taken over the responsibilities of tech lead and am working more with @upamune. During university, I worked part time at an IT company near the university. We released the tools we developed there on GitHub as open source software (OSS). I later joined the Merpay Code Payment Team as an intern, and I’m still working on that team today.
@mihoyolo: I joined the company in October 2018. At Merpay, I spent about six months in Customer Service (CS)/Ops Planning and was involved in launching the CS Ops Team. After that, I was in charge of merchant ops planning for about six months. I then switched roles and worked as a product manager (PM) for Merchant Solution. I had previously worked as an engineer, undertaking SI, consulting, and business development for financial institutions.
At present, I’m working on all sorts of things to ensure the products that I am in charge of provide value to customers and merchants. This includes defining requirements, setting specifications, measuring effect, coordinating with other teams, and formulating roadmaps.
— What does the Code Payment Team do?
@upamune: We are responsible for overall development related to Merpay’s code payment services. One example is the CPM payment service.
Consumer Presented Mode (CPM). With this method, customers pay by presenting a barcode or QR code at the merchant, which is then scanned by the merchant’s barcode/QR reader.
I still feel a personal investment in the CPM payment service, which Merpay launched in March 2019. It was initially introduced at major convenience stores. I remember that immediately after it was launched, some people went to a convenience store to try to make a payment, and that feeling of completing their first payment was amazing. Everyone got excited and shared that excitement over Slack.
— Have you faced any difficulties during the development process?
@upamune: When I first joined the company, the only microservices were authentication and payment infrastructure. Then numerous other microservices were launched all at once, which made coordination between the various teams very difficult. People had to actively get involved in the development of microservices that they were not responsible for. Because the code payment service is the most outward oriented of the backend services, there are many opportunities to use services developed by other teams. Therefore, if there were any missing functions, we would have to go outside the team to get them implemented
@ktr: This team is also responsible for the MPM payment service.
Merchant Presented Mode (MPM). With this method, the customer scans the code presented by the merchant in order to complete payment.
The MPM payment service was launched in June 2019 (FYI: Merpay Places QR Code at Stores and Starts to Handle “Customer Read Method” Code Payment(Available only in Japanese)). The CPM payment service is targeted to large member stores, but the MPM payment service is targeted to small- to medium-sized ones. I joined the team in May, and at that time, the design phase was already completed. We were fully focused on development as the launch approached.
— Have you had similar difficulties with other development projects?
@ktr: It was difficult because there was such little buffer. Even so, I think that development moved forward relatively smoothly because everyone on our team is so talented. The MPM payment service, however, involves printed QR codes, meaning the payment service would be available as soon as the QR codes were mailed to merchants. Therefore, we had to stick closely to the schedule.
The QR code payment service was developed based on various JPQR standards, but we figured out that those specifications did not meet all of Merpay’s needs and had to change them mid-project. It was really a challenge to design a new QR code system at that time.
— Any other projects you are working on?
@ktr: One project involves collaboration with Docomo’s payment service "d barai" and enabling each payment application to read the other’s QR code. This originally was the Mobile Payment Alliance (MoPA) project, an attempt by various payment application companies to create common QR codes. Even though the project stalled for several reasons, including payment business operators merging with each other, we are working with Docomo to once again push the project forward.
— Did you face any difficulties with that?
@ktr: When linking systems between companies, you have to agree on common API specifications, coordinate schedules, and maintain close communication. The two systems, d Payment and Merpay, communicate with each other regarding payments, and synchronizing all the information takes time, which reduces availability. Therefore, we sped up processing and maintained high availability by incorporating parts that could be handled asynchronously.
Creative Develop Solutions to Maintain Robustness and Availability
— What were you careful about when developing for Merpay?
@mihoyolo: It should go without saying that customers need to be able to use Merpay to pay at stores without any problem. This led us to focus on seamlessly providing a robust payment service with high availability. In addition, as we expand the business, we are gradually undertaking new development to improve the experience for both customers and merchants.
— Were there any points that you were careful about in order to maintain robustness and availability?
@upamune: One of the unique aspects is the use of canary releases. With this release method, when a new function is introduced, it is deployed on a single server (this is called the canary server), and after confirming that there are no problems, the function is deployed to other servers. If there are any problems, the whole system can be rolled back by simply restoring the canary server, which leads to greater system stability.
We were also careful to ensure the idempotency of processing (the same payment is not processed multiple times). Because Merpay employs a microservices architecture, problems can arise in various places. We have established systems so that if a problem arises, the process is retried as much as possible, as much as possible. Without guaranteeing idempotence, the same payment could be processed a second time when the process is retried. Therefore, we develop the system to ensure that the service is safe to retry processes. For example, with operations such as Get or Create, if the calling entity lacks the resources, it creates the resources and then returns them when completed.
— Although it is probably impossible to avoid all problems, is there anything you are working to address this?
@upamune: For example, the system is such so that if a problem is detected with part of the microservice related to payments, and the incident makes payment impossible, the service automatically obscures the payment code on the app to prevent the user from making a payment. Of course, we take into account how the merchant wants to handle this. We achieved this by introducing a circuit breaker into the part where the code payment system submits a request to the core system. With a circuit breaker, it is possible to set it so that a request times out if the payment fails five times in a row, and no more requests are sent. In addition, by providing an API that retrieves the state of the circuit breaker (whether payments are possible or not), it is possible to automatically switch between displaying or not displaying payment codes based on this.
Additionally, there is a system so that when there is a problem, it is possible to retrieve related microservice data from Big Query and generate a report by filling out a special form. In this way, we can immediately provide sales staff with information on the scope of the impact on merchants.
— If there are any services or tools that you often use in development, what are they?
@ktr: Our team uses the following services and technologies. These are technologies that all Merpay backend staff use.
As for unique ones that the Code Payment Team uses, there are many that Microservices Platform has developed. For example, there are systems that increase the efficiency of the team’s microservice development and ones that dramatically improve completing and maintaining Kubernetes manifests.
We also use merpay-codepayment-tools, a group of tools for semi-automating operations. For example, we use them for automating releases and various other purposes, such as extracting impact data, generating QR codes, and registering payment processors. Because merpay-codepayment-tools are important tools, they are regularly subject to maintenance.
Desire to Create an Unbelievably Seamless Payment System
— What do you think will happen in the future?
@mihoyolo: I want to transform the payment experience into something completely unanticipated. My hope is we can make it so payments are easy and provide peace of mind, enabling both the stores that offer our service and the customers who enjoy our service to simply focus on the service itself. People only deal with money because it’s a necessary inconvenience. I want to reduce the hassle of paying as much as possible. I think this is what will lead to the “seamless society” that Merpay envisions.
I want to make Merpay even more convenient so that we can achieve that. We can use utility and UX to eliminate that friction.
@upamune: When we launched the CPM payment service, there were some aspects of payment practices and systems that we did not fully understand. Looking back at it now, there are some pieces of the outward-facing API that we should have done differently. Although we can’t make these changes immediately because they involve development with outside companies, I want to make improvements in the medium and long term.
@ktr: I want to provide an experience in which customers have no problems when making payments. And if an error occurs, they should also be able to troubleshoot the issue without knowing the cause exactly. I would like to provide a smoother payment experience by solving those issues.
Type of Person the Code Payment Team Needs
— What are the good points about working at Mercari?
@upamune: Information is open. You have access to almost any information you are interested in. At the weekly all-hands meeting, participants can directly question corporate officers. Another good point about Mercari is its transparency, as even its OKR discussions are made public.
@ktr: The Mercari Group’s values of Go Bold, All for One, and Be a Pro have really taken root within the company, and it is possible to work with talented people who truly embody those values. The Group offers a wide range of benefits, and systems have been introduced that limit downside risks, making it easy to work here.
@mihoyolo: There are members who, regardless of position, will discuss issues until an understanding is reached and can work with an All-for-One spirit once there is agreement. Furthermore, the management team never neglects changes that improve the organization, and they promote diversity and inclusion throughout the organization.
— What type of person is best suited for the Code Payment Team?
@upamune: I think people who can take the initiative and like technology fit well into the team. There is a tech leader, but once the framework is set, it is left up to team members to implement it. Therefore, it is probably not good for passive people. People are always talking about technology. There is even a weekly tech book club and tech chat. People who like technology will fit in.
@ktr: A distinguishing aspect of payment systems is that there are many connections to outside systems, and there is a strong need for the system to be reliable, available, and consistent. People who are interested in developing and operating such systems are a good match for the team.
@mihoyolo: Probably people who understand the systems underlying payments and who can explain them in simple terms to others. Furthermore, people who are particular about providing a better customer experience and merchant experience and are committed to development would fit into the team.
Hiring Information
The Merpay Code Payment Team is currently looking for new members. If you are at all interested in joining us, please take a look at the application guidelines on Mercari Careers.
- Software Engineer, Backend – Merpay
- Product Manager – Merpay (Available only in Japanese)
