This post is for Day 12 of Merpay Advent Calendar 2023, brought to you by @Liu from the Mepay Growth Platform team.
Introduction
You might have noticed that Mercari has started a LINE official account from 9/19 and we have more than 3,500,000 friends as of 2023/12.
After adding the Mercari official LINE account as a friend, customers can link their LINE account with Mercari account for a more personalized experience. Depending on whether they’ve linked their accounts, customers get different kinds of customized messages and rich menus that suit their preferences.
In this blog post, we’ll take a closer look at the LINE Messaging API, the powerhouse behind these interactions, and dive into the behind-the-scenes challenges faced by our backend engineers while using it.
Overview
Let’s begin by understanding how we engage with LINE and where we apply the LINE messaging API in our interactions.
Our marketers connect with LINE customers using two main avenues: Braze, a third-party platform, and the LINE official account manager, LINE’s own management platform.
We maintain both routes for distinct advantages. Through Braze, we can precisely target audiences based on in-app behavior, such as recent Mercari usage or a purchase within the last month. This route also allows us to send customized messages tailored for customers who have linked their Mercari accounts.
On the other hand, the LINE official account manager route enables us to target audiences using LINE-specific information like age, gender, or interaction with previous LINE messages. The customer-friendly interface here also makes it easier for marketers to configure message layouts.
Now, why do we need an integration system in the Braze route? While Braze offers webhook templates for direct LINE messaging API usage, we’ve chosen not to utilize them for two key reasons. Firstly, for privacy concerns, we avoid uploading our customers’ LINE IDs to a third-party platform. Additionally, when sending customized messages, such as recommending items based on customers’ saved search conditions or browsing history on Mercari, direct interaction with Mercari microservices is necessary. This interaction isn’t feasible using Braze alone.
An example of customized message based on customers’ saved search condition:
Transitioning from our integration system to LINE, the LINE messaging API takes center stage. This API facilitates message delivery, rich menu switches, audience management, and various other functionalities.
Messaging – Cloud Functions & Pub/Sub
Our first challenge involves implementing effective flow control when utilizing the LINE messaging API for message delivery.
Our LINE integration system encompasses two messaging scenarios: proactive and reactive. For now, we’ll delve into proactive messaging, where marketers coordinate campaigns through Braze.
In this scenario, our backend engineers grapple with the task of efficiently handling large-scale LINE message deliveries. Initiating a campaign on Braze can result in a sudden influx of millions of requests, presenting a significant challenge. It’s crucial to manage this surge, considering the diverse rate limitations imposed by the LINE API and various Mercari services. Failing to adhere to these limits could trigger a cascade of errors or potentially lead to a service outage.
To overcome this challenge, we’ve established a robust system utilizing cloud functions and Pub/Sub events for processing webhook events. Initially arriving as HTTP requests from Braze, these webhook events undergo a transformation into Pub/Sub events. Cloud Function, with unlimited autoscaling support, ensures smooth operations even during high traffic from Braze. Line workers then pull Pub/Sub messages, allowing precise control over the number of messages processed per second and facilitating effective flow control.
This transformative process guarantees seamless flow control, enabling us to efficiently manage the diverse rate limits.
While the reactive scenario is set for release next year, we anticipate applying similar techniques. The event-to-Pub/Sub transformation remains crucial for sustaining effective flow control. Looking ahead to potential chatbot functionalities, especially those based on Language Model (LLM) technology, this finely tuned flow control mechanism becomes even more critical due to the stricter rate limitations of the OpenAI endpoint. Achieving a balance between timely responses and deferred processing is key to optimizing our interaction capabilities with customers.
Audience Group – Spanner & Cron Worker
The second challenge involves an advanced flow control mechanism utilizing Spanner and a cron worker.
Before delving into the details, let’s provide some context. LINE’s narrowcasting feature enables the selection of audience segments for message casting. For customers who haven’t linked their LINE account and Mercari account, we utilize an audience group comprising all linked customers and perform a reverse selection for effective narrowcasting. This narrowcasting is scheduled and executed using the LINE official account manager, while our LINE integration system manages the audience group using LINE messaging API.
When launching campaigns to encourage customers to link their LINE Mercari accounts, peak linking times result in heightened incoming traffic for our system. This surge triggers a sequence of processes, each demanding meticulous attention. The system executes various actions, such as sending greeting messages, switch rich menus and adding customers to an audience group.
However, as we manage outgoing traffic, it’s imperative to operate within the defined rate limits. Specifically, we adhere to a rate limitation of 60 requests per minute for adding audience members and 2,000 requests per second for sending messages.
To manage rate limitations on the endpoint for adding audience members, a crucial element in customer linking, we’ve established an advanced flow control process. Real-time welcome messages and rich menu switches are promptly dispatched, while customer additions to the audience group are systematically orchestrated by a separate worker with cron tasks running inside.
This meticulous approach ensures not only real-time customer engagement but also compliance with stipulated rate limitations, maintaining a harmonious system operation. Introducing a Spanner table between the worker overseeing linkage events and the cron worker adds an extra layer of control and organization.
The Spanner table schema is as follows:
CREATE TABLE LineLinkageUsers (
ID STRING(36) NOT NULL,
LineID STRING(64) NOT NULL,
LineAudienceStatus INT64 NOT NULL,
Created TIMESTAMP NOT NULL OPTIONS (
allow_commit_timestamp = true
),
Updated TIMESTAMP NOT NULL OPTIONS (
allow_commit_timestamp = true
),
) PRIMARY KEY(ID);The linkage worker inserts a record into this Spanner table when a customer links their accounts. The cron worker retrieves customers not yet added to the audience group from the Spanner table by the LineAudienceStatus, calls the audience adding API every 3 minutes for a group of customers, and updates the LineAudienceStatus to flag the customers as added.
Another critical aspect of this process is acknowledging the existence of a maximum number of concurrent operations, precisely set at 10 for the audience adding endpoint. This consideration becomes particularly significant when there’s an abundance of customers in the requests over the past 10 calls. While the actual addition occurs seamlessly in the background on the LINE server, the potential for encountering 409 errors arises if 10 concurrent adding processes are in progress.
To address this challenge, potential solutions involve limiting the maximum number of customers added in each request or incorporating a confirmation step before each request. This confirmation step, executed using the endpoint GET https://api.line.me/v2/bot/audienceGroup/{audienceGroupId} provided by LINE, allows for a proactive check on job process status. By ensuring there are no more than 10 requests already running, we can effectively prevent encountering 409 errors.
Conclusion
In conclusion, our journey through Mercari’s LINE integration challenges sheds light on the crucial role of flow control. By establishing a connection between Braze and the LINE official account manager, we enhance customer engagement through personalized messaging. Exploring the intricacies of the LINE Messaging API, we highlight the need for a dedicated integration system. In proactive messaging, our robust system efficiently handles large-scale LINE message deliveries, ensuring compliance with various rate limitations. The Audience Group management, utilizing Spanner and Cron Workers, adeptly manages peak traffic during customer linkage. As we anticipate the release of reactive scenarios and potential chatbot functionalities, maintaining effective flow control remains vital. In navigating these challenges, Mercari continues to optimize customer experiences through a seamless LINE integration.
Thank you for taking the time to read until the end. Here is a QR code to add friends to our official LINE account now! 😀
Tomorrow’s article will be by @abcdefuji. We hope that you are looking forward to it!
