Mercari Engineering blog

Enabling AI usage at Mercari with Secure Devin Management

Fri, 03 Apr 2026 10:25:21 GMT

Introduction

Hello. I am @hi120ki, an AI Security engineer at Mercari.

At Mercari, we have rolled out Devin, an AI Agent service, to multiple teams across the company. Devin is a service that can autonomously investigate code, write code, and submit pull requests. However, operating it at an organizational level comes with several management challenges.

In this article, I will introduce how the AI Security team worked together with the AI Agent Platform team to scale the operation of Devin across the entire organization, by building a custom Terraform provider and a set of automated management tools, all powered by the Devin Enterprise API. Through these tools, we established mechanisms for member and permission management, secret rotation, API key lifecycle management, and auditing. We hope that this serves as a blueprint for securing deploying and operating Devin across an enterprise.

Challenges of Enterprise Operations

At Mercari, we use Devin’s Enterprise plan. To operate an AI Agent running in a remote environment at an organizational scale, SSO through Okta, audit logs, permission management, and environment isolation per team were essential requirements, which led us to choose this plan.

In Devin Enterprise, rather than sharing a single Organization as with the Core or Team plans, multiple Organizations are centrally managed through an Enterprise management layer. Mercari has numerous teams spanning multiple business domains, and the information each team handles must be kept isolated and protected. For this reason, we assign Organizations according to team or purpose.

However, in an environment with more than 10 Organizations and a large number of users, the following challenges arise.

Challenges in Permission Management

Assigning members to Organizations relies on manual operations
Tracking the state of "who belongs to which Organization" is difficult

Challenges in Secret Management

Authentication credentials for each third party service must be configured individually per Organization
Rotating secrets manually across all Organizations is time consuming

Challenges in Access Control

Devin does not offer expiration management for API keys as a standard feature, creating a risk of long lived, unrotated API keys remaining in each Organization

As Devin adoption grows, the number of Organizations to manage increases, and the burden of these challenges grows with it. Previously, we relied on manual operations through the Web UI, but since late 2025, Devin has released Enterprise API v3, making it possible to automate most management operations through the API. In response, we have built an in-house management platform using Go and GitHub Actions.

Overview of the Devin API

Devin provides its latest Enterprise management API as v3. It allows management of Members, Roles, Secrets, and Knowledge at both the Enterprise and Organization levels. Using the v3 API, we have built the following automated management capabilities:

Custom Terraform provider
Bulk secret rotation
Google Cloud service account key rotation
Integration with the security management platform

Only API key management uses the v2 API. The v2 API allows creation, retrieval, and deletion of API keys across multiple Organizations, and we use it for the following:

Periodic invalidation of API keys issued by users
API key management for internal Agent access to Devin Wiki

These API specifications are documented as REST-format APIs in Devin’s official documentation, complete with detailed specifications for requests and responses, and each function can be invoked by implementing a standard REST API client. For this implementation, we built these REST API clients using Go, which is widely used within Mercari, organizing them so that each API corresponds to a function, making them easy to reuse.

The following sections describe each of these capabilities in detail.

1. Custom Terraform Provider

The core of our management platform is Organization and member management through a custom Terraform provider built with the Terraform Plugin Framework.

At Mercari, Terraform is the standard tool for Infrastructure as Code (IaC) resource management, including Google Cloud, and engineers work with it on a daily basis, which is why we chose this approach. Managing Devin through IaC allows us to insert PR reviews into member additions and permission changes, and makes the state of Organizations and members visible in code. Since no official Terraform provider is available at this time, we built our own.

Users and administrators define each team’s Organization in Terraform. ACU (Agent Compute Unit) limits are also set here to control usage per team. max_cycle_acu_limit sets the overall ACU cap for the Organization, and max_session_acu_limit sets the cap per session, preventing unexpected cost overruns.

resource "devin_organization" "mercari_example_team" {
  name                  = "mercari-example-team"
  max_cycle_acu_limit   = 500
  max_session_acu_limit = 250
}

Member assignments to Organizations are also managed declaratively through Terraform.

# Member definition (referenced by email address)
data "devin_member" "mercari_example_team" {
  for_each = toset([
    "user-1@example.com",
    "user-2@example.com",
    "user-3@example.com",
  ])
  email = each.value
}

# Assignment to Organization
resource "devin_organization_member" "mercari_example_team" {
  for_each = data.devin_member.mercari_example_team

  user_id     = each.value.user_id
  org_id      = devin_organization.mercari_example_team.org_id
  org_role_id = "mercari_org_member"
}

Adding Organizations, changing ACU limits, and adding or removing members all follow the standard development flow of modifying Terraform code, reviewing a PR, and merging. The output of terraform plan clearly shows who will be added to or removed from which Organization, preventing unintended permission changes.

This Terraform provider also manages Devin Knowledge. Knowledge functions similarly to Agent Skills within Devin. In Mercari’s Devin environment, each team is separated into different Organizations and cannot see each other’s usage. While this isolation is desirable from a security standpoint, it makes sharing practical know-how difficult. By making Knowledge manageable through the provider, we enabled the distribution of practical knowhow across teams.

2. Bulk Secret Rotation

Devin launches an independent virtual machine for each Session, so in its initial state, it only has permissions for source code management services such as GitHub. Connecting to cloud environments or ticket management services requires configuring authentication credentials such as API keys individually.

At the same time, as an AI Agent, Devin can freely use any API keys it is given, and members within an Organization can access the file system and shell inside Sessions. This means credentials must be handled with care. At Mercari, we centrally manage the API keys configured in Devin and rotate them at short intervals, ensuring that long lived credentials do not remain on Devin.

However, manual rotation is a heavy burden. Previously, rotating multiple Secrets across numerous Organizations consumed a significant amount of time. When Devin added Secret management to the v3 API in January 2026, it became possible to automate these operations. The current rotation procedure is as follows:

The Devin administrator rotates credentials in each respective service
The new credentials are added to a previously created Google Cloud Secret Manager
The automation is triggered through GitHub Actions
Rotation is executed, distributing secrets from Secret Manager to each Organization

This allows us to fully automate secret rotation, without any additional effort if new Organizations are created.

3. Google Cloud Service Account Key Rotation

At Mercari, we primarily use Google Cloud, and to retrieve libraries and connect to test environments, we need to grant Google Cloud permissions to Devin. However, Devin currently does not have an OIDC token issuance feature that would allow it to work with Workload Identity Federation, so we must use service account keys.

However, Mercari follows Google Cloud’s official best practices and prohibits the issuance of service account keys across the board through Organization Policy. Therefore, we set up a dedicated Google Cloud Project or Devin excluded from the Organization Policy, with iam.serviceAccountKeyExpiryHours as a compensating control. This ensures that even if automation stops, service account keys are automatically disabled after a fixed period.

On top of this framework, we periodically rotate and assign individual service account keys for each Organization.

4. Integration with the Security Monitoring Platform

One of the requirements for adopting Devin Enterprise was audit logging. At Mercari, Anna from AI Security and Threat Detection and Response team had built integration with our in house security monitoring platform through the Devin v3 API.

We are using an enterprise-level service user with admin access and the Enterprise Audit Logs endpoint which – unlike the v2 endpoint – has pagination. We have a Google Cloud’s Cloud Run Job that pulls and forward all the new logs to a PubSub topic. From there we can analyze the logs and store them in BigQuery for investigation purposes.

5. Periodic Invalidation of API Keys Issued by Users

We enforce API key expiration, by an automation that retrieves all API keys across the entire Enterprise and automatically invalidates any keys that have exceeded a certain period since creation. Devin currently does not provide this as a standard feature.

These API keys are primarily used to connect to Devin MCP. Since source code can be obtained indirectly through API keys, strict management is required. In development environments where multiple AI Agents are in use, situations can arise where credentials remain in configuration files of unused Agents, or where someone sets a personal API key in a custom Agent shared with colleagues and publishes it within the company.

By automatically invalidating API keys after a certain period, we maintain a state where only actively used Agents hold API keys. For Agents shared among multiple people, we have them use API keys managed through Google Cloud Secret Manager as introduced in the next section. This also achieves visibility into the permissions held by each Agent.

6. API Key Management for Internal Agent Access to Devin Wiki

At Mercari, we operate a separate Organization dedicated to Devin Wiki, apart from the development Organizations for each team. Devin Wiki allows retrieval of repository contents and natural language search through Devin MCP.

When an AI Agent directly performs source code exploration, it consumes a large amount of context. By delegating source code investigation to Devin in situations where it is needed, context consumption can be reduced.

However, using Devin MCP requires an API key, and as described in the previous section, keys are automatically invalidated after a certain period. While it is possible to create exception API keys, this cannot completely prevent misuse. Therefore, we built automation that periodically recreates API keys at short intervals and stores them in Google Cloud Secret Manager.

This enables us to centrally manage the service accounts of AI Agents using Devin MCP through Terraform, providing visibility into usage, while also preventing misuse through periodic key recreation.

resource "google_secret_manager_secret" "shared_wiki_api_key" {
  secret_id = "shared-wiki-api-key"
}

resource "google_secret_manager_secret_iam_member" "shared_wiki_api_key" {
  for_each  = toset(local.accessor_service_accounts_shared_wiki_api_key)
  secret_id = google_secret_manager_secret.shared_wiki_api_key.secret_id
  role      = "roles/secretmanager.secretAccessor"
  member    = "serviceAccount:${each.value}"
}

locals {
  accessor_service_accounts_shared_wiki_api_key = [
    "agent-1@---.iam.gserviceaccount.com",
    "agent-2@---.iam.gserviceaccount.com",
  ]
}

CI Pipeline for Unified Orchestration

All of these management operations are automated through GitHub Actions. When building custom management tools for SaaS administration, long term maintenance is unavoidable. Considering handoffs during organizational changes, it is necessary to keep dependencies small and choose technologies and platforms that are easy to maintain.

While Secret Manager and service accounts reside on Google Cloud, we chose GitHub Actions for execution. Since automation within the repository runs directly without deployment, maintenance effort is reduced. By not holding unnecessary cloud resources, we also keep costs low and reduce the mental burden during management and handoffs. In addition to scheduled runs, we support manual triggers (workflow_dispatch), allowing immediate secret rotation in emergencies.

On the other hand, because GitHub Actions can be freely executed, we strictly configure permission management and branch protection settings. For credential retrieval, we use Google Cloud Workload Identity Federation to securely access service accounts and Secret Manager from GitHub Actions.

Conclusion

In operating Devin Enterprise at scale, we supplemented management requirements that could not be covered by standard features alone with custom tools built using the v2 and v3 APIs. This helped us overcome management challenges that had previously relied on manual work, enabling us to provide and properly manage many Organizations in parallel.

The currently available Devin v3 API already includes the endpoints required for Enterprise administration. Going forward, we plan to continue automating the safe management of a wider range of resources as Devin’s capabilities expand.

We hope this article will be helpful to those facing similar challenges.

If you are interested in AI and LLM adoption and security initiatives at Mercari, please visit Mercari’s careers page.

Legacy Image Provider to Cloudflare Images: Traffic Estimation and Safe Rollout

Thu, 02 Apr 2026 09:00:26 GMT

Abstract

Retiring a legacy image resizing path sounds straightforward until you realize how many “invisible” callers exist: long-lived app versions, embedded clients, partner integrations, and bots. In our case, Mercari Platform Network team, migrated a legacy image transformation pipeline to Cloudflare Images while keeping existing URLs working.

This article focuses on the Cloudflare setup and safe rollout and explains the tradeoffs we made to reduce risk. It intentionally skips backend service details so the story stays centered on edge configuration, traffic estimation, and operational safety.

What the migration looked like

This migration had one non-negotiable constraint: existing image URLs had to keep working throughout the transition. That single requirement shaped almost every design decision, because it forced us to run old and new paths side by side and prove safety with production traffic.

The diagram below shows the simplified request flow. The important point is that the “easy” part is adding a new provider, while the hard part is understanding how many request patterns exist in the wild and how they interact with caching.

At first glance, this kind of migration can look like a straightforward origin swap. That can be true for a small system with one caller and short cache lifetimes, but it is rarely true for a long-lived public image pipeline.

In practice, we had to account for compatibility with existing request patterns, unexpected side effects, operational costs, zero-downtime rollout requirements, and monitoring that could catch regressions quickly. Those concerns became concrete challenges once we started mapping dependencies.

Migration challenges

A legacy image pipeline tends to sit at the boundary between many systems. Even if the official callers have moved on, old patterns can survive for years through caches, bookmarks, copy-pasted snippets, and client code that is hard to update.

That creates a specific failure mode: the traffic volume may look small, but the blast radius can still be large. When something breaks, it often breaks in places that are difficult to reproduce in staging.

In practice, this means migrations like this are less about “switching a backend” and more about dependency discovery. If we miss a dependency, we learn about it in production, and usually at the worst time.

Direct Amazon S3 access from Cloudflare

We expected that “just point Cloudflare to the bucket” would be the simplest approach, but a seemingly minor naming choice became a TLS constraint. Our goal was to keep an HTTPS (HTTP over TLS) path from the edge to the origin while preserving a legacy bucket name.

Amazon Simple Storage Service (Amazon S3) provides several ways to access a bucket and its contents. In many systems, any of these options can work, but the details matter once you require HTTPS end-to-end.

Type	How to access	Restrictions
Path-style	https://s3-ap-northeast-1.amazonaws.com/your-bucket-name/your-bucket-contents	Not recommended by AWS. It had a deprecation plan, but it is still active.
Virtual host-style	https://your-bucket-name.s3-ap-northeast-1.amazonaws.com/your-bucket-contents	If the bucket name is `xxx.xxx.com`, HTTPS can break because of certificate mismatch.
Host header-style	https://s3-ap-northeast-1.amazonaws.com/your-bucket-contents -H “Host: your-bucket-name”	You must inject the `Host` header somewhere along the request path.

Mercari’s S3 bucket for images has a long history, and it uses a legacy naming convention based on a domain name like xxx.mercari.com.

Cloudflare also provides Cloud Connector, but it did not work cleanly for this scenario. When we used a bucket name like xxx.mercari.com, we hit an invalid SSL certificate (Error code 526) during the HTTPS handshake.

Egress traffic and image quality and cost

Even if every request succeeds, changes in compression and resizing behavior can change egress volume and affect image quality. If we shipped that kind of change blindly, we could create a cost regression or a customer experience regression without any obvious outage.

With Cloudflare Images, we expected changes in output size and compression behavior, which directly affects egress traffic. For example, if the average image size increases by 50%, egress typically increases by roughly 50% as well.

At the same time, if images look different or lose important details compared to the legacy image provider, the migration can negatively impact customer experience. That meant we had to measure both size and perceptual similarity before increasing rollout.

Zero-downtime rollout

Even with correct edge configuration, rollout mechanics can still break production if we ramp too quickly or if we misjudge cache behavior. We assumed that unknown legacy access patterns would exist, and we designed the rollout so failures would stay small, measurable, and reversible.

Several factors could block or slow down the release process, including S3 rate limits, cache rebuilding, and unknown legacy access patterns. As a result, we treated rollout design as a first-class engineering problem rather than a final deployment step.

How we resolved it: S3 access from Cloudflare

This section explains the concrete edge configuration that allowed Cloudflare to fetch from S3 over HTTPS even with our legacy bucket name. The key idea was to use an origin override so we could keep the request URL stable while controlling the origin host and headers.

Because virtual host-style access did not support HTTPS for our bucket name and AWS discourages path-style access, we chose the host header-style approach for this migration. That allowed us to connect to the regional S3 endpoint while presenting the legacy bucket name in the Host header.

Using our internal Terraform module cdn-kit, we implemented this by routing S3 origin access through an origin override. We also separated the “real” public endpoint from a placeholder endpoint used only for origin modifications, so we could keep the rules explicit and auditable.

module "cdn_kit" {
  # ...
  endpoints= {
    "@" = {
      backend = {
        host = "legacy-image-provider-endpoint"
      }
    }
    "s3" = { # placeholder endpoint for origin modification
      backend = {
        host = "s3-ap-northeast-1.amazonaws.com"
      }
    }
  }
  request = {
    origin_modifications = [
      {
        host       = "xxx.mercari.com" # bucket name
        expression = <<EOC
          (not starts_with(http.request.uri.path, "/prefix/xx/"))
        EOC
        origin     = "s3.${var.domain}"
      }
    ]
  }
  # ...
}

This approach kept the migration reversible. If we saw unexpected errors, for example, edge error rate, origin error rate, and origin response time, we could disable the origin modification rule and fall back to the legacy provider without changing client behavior.

How we resolved it: image quality, egress and cost

In this section we explain how we turned the “it might be expensive” fear into measurable signals and concrete guardrails. Instead of guessing, we validated behavior under controlled traffic and used the results to set rollout pacing.

Two areas could be impacted by the migration. First, we needed to confirm whether Cloudflare Images behaved like the legacy provider in terms of resizing and compression.

Second, we needed a way to estimate cost. Cloudflare Images uses a different billing model, so we had to validate how to measure and forecast usage with enough confidence to proceed.

Image quality

Availability metrics cannot detect a silent quality regression, so we validated outputs directly. The goal was to ensure that “successful” responses still delivered images that looked the same to customers.

Cloudflare Images uses a slightly different compression algorithm than the legacy provider. We randomly sampled thousands of image IDs from access logs and compared outputs across parameters like quality and width.

We found that Cloudflare Images often produced larger files, especially for WebP. This could increase egress traffic and cost by up to ~50% in some cases, even though JPEG outputs were sometimes smaller than the legacy provider.

Beyond file size, we compared similarity and pixel-level differences between the legacy outputs and Cloudflare outputs. Pixels differed slightly after resizing, but similarity stayed almost unchanged. Based on this, we chose a lower quality setting than Cloudflare’s default to reduce file size while keeping high visual similarity.

Egress and cost

Cloudflare Images pricing is based on unique transformations, which means the long tail can matter more than request volume. We needed a method that matched Cloudflare’s 30-day counting model, otherwise our estimates would drift.

Because Cloudflare Images uses a 30-day window to count unique transformations, it is hard to estimate monthly usage from per-day or per-hour samples. The safest approach is to run a 30-day query and use the result as the baseline.

SELECT
    APPROX_COUNT_DISTINCT(ClientRequestURI) AS unique_transformation
FROM `...access_logs`
WHERE
    EdgeStartTimestamp BETWEEN TIMESTAMP("YEAR-MONTH-01") AND TIMESTAMP("YEAR-NEXT_MONTH-01")
  AND ClientRequestSource = 'eyeball'
  AND EdgeResponseStatus = 200
  AND REGEXP_CONTAINS(ClientRequestURI, r'^(/prefix-01|/prefix-02|...)')

Cloudflare recently introduced analytics in the dashboard and changed rolling 30-day window to natural month window, which makes ongoing monitoring much easier.

The first day’s number is usually high because the system starts counting unique transformations from a cold state. It typically drops over subsequent days because many accesses have already been counted within the current month window.

How we resolved it: zero-downtime rollout details

In this section we describe implementation details that made the rollout operationally safe.

How can we do the rollout from 0% to 100%?

The key idea was to add an abstract path layer that could route to both the legacy provider and Cloudflare Images while keeping client-facing URLs stable.

That abstraction also made the rollout easier to reason about. By standardizing request patterns early, we could focus measurement on the most important traffic and track what remained for migration.

To build the abstract URL path layer, we used Cloudflare URL rewrite rules to map an abstract prefix to provider-specific paths. This let us switch a controlled slice of requests without requiring clients to adopt a new URL format.

rewrites = [
  # before migration, legacy image provider
  { prefix = "/abstract-01/", target = "/old-prefix/settings/" },
  # after migration, Cloudflare Images
  { prefix = "/abstract-01/", target = "/cdn-cgi/image/settings/" }  
]

To avoid a random split, we increased traffic by matching image IDs using a regex-based rollout. This approach made the rollout deterministic, which improved debuggability and reduced the risk of inconsistent customer experience.

path_regex="/\\d+(000[0-9]{1})_\\d+\\.jpg"   # 0.1%
path_regex="/\\d+(0[0-9]{3})_\\d+\\.jpg"     # 10%
path_regex="/\\d+([0-1][0-9]{3})_\\d+\\.jpg" # 20%
# ...
path_regex="/\\d+([0-7][0-9]{3})_\\d+\\.jpg" # 80%
path_regex="/\\d+([0-8][0-9]{3})_\\d+\\.jpg" # 90%

Cache rebuilding

Cache rebuilding was the main operational risk during rollout. If we rebuilt too quickly, we could overload S3 and create widespread errors that would look like a CDN outage even though the edge configuration was correct.

We used a three-phase rollout. First, we ran a canary at less than 0.1% of traffic to measure cache rebuild behavior and the impact of S3 rate limits, with a focus on non-200 responses.

Next, we gradually increased to 1%, 5%, and 10% while confirming rebuild time, S3 request patterns, and error rates. Once those signals stayed stable, we moved to full rollout and increased by 10% per release until reaching 100%.

Cache purging

Cloudflare Images also uses a different cache purge mechanism. You cannot always purge by the transformed URL. Instead, you purge by prefix using the origin path.

For example, if a resized image is served via /cdn-cgi/images/quality=85/somewhere/imageid, you purge using a prefix that targets /somewhere/imageid on the origin. That means the cache purging system must implement the same mapping.

During our migration, we updated the cache purging system first and only then increased rollout percentage. Since the percentage-based rollout triggers cache rebuilding, it does not rely on per-URL purges during the ramp-up.

Conclusion

We migrated from a legacy image provider to Cloudflare Images by treating dependency discovery as the main work and using traffic estimation to decide when it was safe to take the next step. We avoided the common trap of switching traffic first and learning about breakage later.

If you take one idea from this story, make it this: do not start by switching traffic. Start by learning what you would break, and design your rollout so failures are small, measurable, and reversible.

Safe Chunked Execution for Large-Scale Data Updates and Deletions

Mon, 30 Mar 2026 16:38:44 GMT

I’m taka-h from the DBRE (DataBase Reliability Engineering) team.

Large-scale data updates and deletions can often be expressed straightforwardly in SQL, but executing them all at once introduces significant operational risk. For example, large transactions can cause replication lag, increased database load, and UNDO log bloat — all of which can ultimately lead to service disruptions.

To address this, we implemented a general-purpose tool that lets you describe the operation you ultimately want to perform — such as an UPDATE or DELETE — in a SQL-like syntax, while automatically splitting execution into safe, manageable chunks at runtime. The tool also incorporates the operational controls that real-world use demands: the ability to adjust settings like processing speed while a job is running, and the ability to automatically pause based on monitoring results.

In this article, we explain why this problem occurs, how we have historically worked around it, and how our new tool achieves both safety and operational manageability. At the end, we also publish the tool’s README, which should serve as a starting point for anyone facing similar challenges who wants to implement something tailored to their own environment.

Note that this tool is designed to support the following database operations within our organization:

Archiving or deleting data
Backfilling data
Bulk updating data

Challenges with Large-Scale Data Update and Delete Operations

With small databases, running the target SQL directly may not cause any issues. However, when dealing with data beyond a certain scale, executing that same SQL as a single bulk operation becomes a risk in itself.

The primary reason is that processing a large number of rows tends to produce large transactions, whose side effects can ripple across the entire database. Specifically, this can cause delays in change propagation (such as replication lag), increased database load, and UNDO log bloat that impacts recovery and overall performance.

The traditional approach to handling these situations has been to "process the target data in smaller pieces." In practice, this meant asking engineers to write SQL that splits the target rows by primary key into manageable batches and executes a series of short transactions, or preparing a one-off dedicated script each time the need arose.

BEGIN;
-- Processing rows by specifying a small range of primary keys at a time
DELETE FROM items WHERE id IN (...);
COMMIT;
SLEEP ...;

However, writing a one-off script every time, or manually extracting and splitting target primary keys, is tedious. It also places the burden on the requester to construct SQL in a "safe" manner, and these operational costs add up over time.

To address this, we implemented a tool that provides a general-purpose solution to the problem.

Solution: A General-Purpose Tool

With this tool, users describe their intent as "the condition they ultimately want to satisfy" in a SQL-like syntax. At runtime, the tool fetches the rows matching that condition by primary key, splits them into batches, and repeatedly executes short transactions — allowing UPDATE and DELETE operations to proceed safely.

In practice, situations such as high database load or unexpected issues can arise independently of the progress of the deletion or update job itself. For this reason, it is important to be able to adjust processing speed and behavior on the fly, and to automatically pause execution when necessary.

To meet these requirements, the tool supports changing settings such as processing interval and batch size while a job is running. This draws on the same philosophy that makes gh-ost — MySQL’s online schema change tool — operationally convenient: the ability to control execution while it is in progress. The tool also incorporates a mechanism to automatically pause processing based on monitoring results.

The final configuration example is shown in the diagram above. It allows you to separately configure what you want to execute (described in a SQL-like syntax) and how to execute it safely (operational concerns). Additionally, most of the settings under processing can be changed while the job is running.

This tool was primarily implemented with the help of generative AI, has been verified to work correctly, and is already in use internally. While we ultimately decided not to release the source code itself as OSS, we will publish the tool’s README.md in the next section. We hope that by adapting the requirements to your own environment and using generative AI, you will be able to build and use a similar tool yourself.

If you find it useful or have ideas for improvement after trying it out, we would love to hear your thoughts — feel free to discuss them on social media or elsewhere. We would also appreciate it if you spread the word by mentioning that you built it using the README.md published by Mercari’s DBRE team.

Finally, Mercari is currently hiring an Engineering Manager (EM) for the DBRE team, which the author of this article belongs to. Please see here for more details.

README.md for the General-Purpose Data Update Tool


# data-updater

A tool for batch data operations (UPDATE, DELETE, or NULL) on database records using primary keys with configurable conditions.

## Features

- Cursor-based batch processing with configurable batch size
- **Three operation types**: UPDATE, DELETE, and NULL (before_sql only)
- **Parallel execution**: SELECT and UPDATE operations run concurrently for better performance
- **Replica support**: Route SELECT queries to replica database to reduce primary load
- **JOIN support**: Complex queries with multiple tables to identify target records
- **Before SQL hooks**: Execute SQL before each batch (archiving, audit logging)
- **Custom ORDER BY**: Process records in custom order
- Interactive commands for runtime control (similar to gh-ost)
- **YAML-based configuration**: All settings in a single configuration file
- Real-time status monitoring with ETA
- Pause/resume functionality
- Dynamic configuration updates
- Socket-based remote control interface
- **Failed ID tracking**: Records failed updates and displays summary on exit
  - For batch-level failures: Records only first and last ID of the failed batch
  - For partial updates: Logs the discrepancy but doesn't track individual IDs
  - Writes detailed report to file if >100 failures
- **Automatic resume**: Saves progress to status file after each batch
  - Automatically resumes from last successful position on restart
  - No need to manually track progress or specify resume points
  - Status files are adapter/table specific for multiple concurrent jobs

## Install

```bash
go install github.com/xxx/cmd/data-updater
```

## Quick Start

1. Create a configuration file:

```yaml
# config.yaml
database:
  host: localhost
  port: 3306
  user: myuser
  password: mypassword
  database: mydatabase
  options:
    charset: utf8mb4
    parseTime: "true"

processing:
  batch_size: 1000
  interval: 1s

adapter:
  table_name: users
  pk_columns:
    - user_id
  update_sql: "status = 'processed', updated_at = NOW()"
  where_clause: "status = 'pending'"
```

2. Run the tool:

```bash
# Normal mode - executes updates
data-updater --config config.yaml

# Debug mode - SELECT only, no updates
data-updater --config config.yaml --debug

# Resume from specific ID
data-updater --config config.yaml --resume-from "12345"

# Show version
data-updater -v
```

## Operation Types

The tool supports three operation types:

### UPDATE (default)
Updates records matching the specified conditions.

```yaml
adapter:
  table_name: users
  pk_columns: ["user_id"]
  operation: update  # or omit (default)
  update_sql: "status = 'processed', updated_at = NOW()"
  where_clause: "status = 'pending'"
```

### DELETE
Deletes records matching the specified conditions.

**Important**: The DELETE operation permanently removes data. Always test with --debug mode first.

```yaml
adapter:
  table_name: old_logs
  pk_columns: ["id"]
  operation: delete
  where_clause: "created_at < '2023-01-01'"
```

### NULL
Executes only before_sql without UPDATE or DELETE. Useful for archiving, copying, or transforming data.

```yaml
adapter:
  table_name: items
  pk_columns: ["id"]
  operation: "null"
  before_sql: |
    INSERT INTO archived_items (id, name, created_at, archived_at)
    SELECT id, name, created_at, NOW() FROM items WHERE id IN (?)
  where_clause: "status = 'inactive'"
```

## Configuration

All settings are managed through a YAML configuration file:

### Database Configuration
```yaml
database:
  host: localhost         # Database host (default: localhost)
  port: 3306             # Database port (default: 3306)
  user: myuser           # Database user (required)
  password: mypassword   # Database password (required)
  database: mydatabase   # Database name (required)
  options:               # MySQL connection options (optional)
    charset: utf8mb4
    parseTime: "true"
    loc: UTC
    timeout: 30s
  # Replica configuration (optional)
  replica_host: replica-db.example.com  # SELECT queries go here
  replica_port: 3306                     # Defaults to primary port
  replica_user: replica_user             # Defaults to primary user
  replica_password: replica_password     # Defaults to primary password
```

When replica_host is configured:
- SELECT queries (fetching PKs, COUNT) are routed to replica
- UPDATE/DELETE operations always use primary
- SELECT FOR UPDATE (pessimistic locking) uses primary

### Processing Configuration
```yaml
processing:
  batch_size: 1000          # Number of rows per batch
  interval: 1s              # Time between batches (e.g., 1s, 500ms, 2m)
  debug_mode: false         # Log queries without executing updates
  pipeline_buffer: 1        # Buffer size for parallel SELECT/UPDATE
  pessimistic_locking: true  # Use SELECT FOR UPDATE (default: true)
  lock_retry_count: 3       # Number of lock acquisition retries
```

### Adapter Configuration
```yaml
adapter:
  table_name: users         # Target table (required)
  table_alias: u            # Alias for main table (required when using joins)
  pk_columns:               # Primary key column(s) (required)
    - user_id
  operation: update         # "update" (default), "delete", or "null"
  update_sql: "status = 'processed'"  # SET clause (required for update)
  before_sql: "..."         # SQL to execute before operation (required for null)
  where_clause: "status = 'pending'"  # Additional WHERE (optional)
  join_clause: "..."        # JOIN statements (optional)
  order_by: "created_at"    # Custom ORDER BY (optional, defaults to PK)
```

### Interactive Control
```yaml
interactive:
  enabled: true             # Enable socket-based control
  socket_path: "/tmp/data-updater.sock"  # Unix socket path
```

### Status File (Automatic Resume)
```yaml
status_file:
  enabled: true             # Enable automatic resume
  path: "/var/lib/status"   # Custom path (optional)
```

## Advanced Features

### JOIN Support

Use JOINs for complex queries that need to reference multiple tables:

```yaml
adapter:
  table_name: items
  table_alias: i
  pk_columns: ["id"]
  operation: delete
  join_clause: |
    LEFT JOIN transaction_evidences te ON te.item_id = i.id
  where_clause: |
    i.status = 'cancel'
    AND te.id IS NULL
```

**How it works:**
1. SELECT query uses JOINs + WHERE to fetch PKs
2. DELETE/UPDATE query only uses primary keys (no JOINs)

### Before SQL (Pre-operation Hook)

Execute SQL before each batch within the same transaction:

```yaml
adapter:
  table_name: items
  pk_columns: ["id"]
  operation: delete
  before_sql: |
    INSERT INTO deleted_item_ids (id, created, deleted)
    SELECT id, created, NOW() FROM items WHERE id IN (?)
  where_clause: "status = 'cancel'"
```

**Notes:**
- Use IN (?) placeholder - expanded to all PKs in the batch
- For composite keys: (col1, col2) IN (?)
- Executed atomically with the main operation
- If before_sql fails, entire transaction is rolled back

### Custom ORDER BY

Process records in a specific order:

```yaml
adapter:
  table_name: items
  table_alias: i
  pk_columns: ["id"]
  order_by: "i.created, i.id"
```

### Understanding update_sql

The update_sql parameter specifies the SET clause. **Do not include trailing semicolons.**

```yaml
# Simple status update
update_sql: "status = 'processed'"
# Results in: UPDATE users SET status = 'processed' WHERE user_id IN (...)

# Multiple columns
update_sql: "status = 'archived', archived_at = NOW()"

# Using CASE statements
update_sql: |
  status = CASE
    WHEN last_login < NOW() - INTERVAL 30 DAY THEN 'inactive'
    ELSE 'active'
  END
```

**Important**:
- Do NOT include UPDATE keyword, table name, or WHERE clause
- The tool automatically adds WHERE pk IN (...) for batch updates

### Using where_clause for Idempotent Operations

Make updates safe to run multiple times:

```yaml
adapter:
  update_sql: "status = 'processed', processed_at = NOW()"
  where_clause: "status = 'pending'"
# Results in: UPDATE users SET ... WHERE user_id IN (...) AND status = 'pending'
```

## Command Line Options

- --config, -c: Path to YAML configuration file (required for operation)
- --debug, -d: Enable debug mode (SELECT only, no updates)
- --resume-from: Manual resume from specific primary key(s)
- --total-rows: Skip initial COUNT query and use provided value (e.g., --total-rows 1000000). Also used as a stop condition based on rows_handled (rows selected), not rows_processed (rows affected by UPDATE)
- --pk-source: Read PKs from file/directory instead of table (local path or gs://bucket/path)
- --version, -v: Show version information
- --help, -h: Show help message

## Interactive Commands

Control the tool via Unix socket:

```bash
# Show status
echo "status" | nc -U /tmp/data-updater.sock

# Pause/resume processing
echo "pause" | nc -U /tmp/data-updater.sock
echo "resume" | nc -U /tmp/data-updater.sock

# Change batch size
echo "batch-size 5000" | nc -U /tmp/data-updater.sock

# Change interval
echo "interval 500ms" | nc -U /tmp/data-updater.sock

# Show help
echo "help" | nc -U /tmp/data-updater.sock

# Auto-interval: show status / enable / disable / set min
echo "auto-interval" | nc -U /tmp/data-updater.sock
echo "auto-interval on" | nc -U /tmp/data-updater.sock
echo "auto-interval off" | nc -U /tmp/data-updater.sock
echo "auto-interval min 200ms" | nc -U /tmp/data-updater.sock
```

## Debug Mode

Debug mode allows you to verify queries without executing updates:

```bash
data-updater --config config.yaml --debug
```

Example output:
```
INFO DEBUG: UPDATE query that would be executed query="UPDATE users SET status = 'processed' WHERE user_id IN (?,?,?)" args_count=3 primary_keys_count=3
```

## Resume Feature

### Automatic Resume (Default)
- Progress saved after each successful batch
- On restart, automatically resumes from last position
- Status files named: data-updater-{table}-{adapter}.status

### Manual Resume
```bash
# Single primary key
data-updater --config config.yaml --resume-from "12345"

# Composite primary key
data-updater --config config.yaml --resume-from "tenant1,12345"
```

### Resume Priority
1. Manual --resume-from (highest)
2. Status file (if exists)
3. Adapter's initial cursor (default)

### Skip COUNT Query
Use --total-rows to skip the initial COUNT query:
```bash
# Useful for large tables or retries where you know the total
data-updater --config config.yaml --total-rows 1000000
```

This is particularly useful when:
- Retrying after interruption (you already know the count)
- Large tables where COUNT(*) is expensive
- Faster startup when exact count is not critical

**Stop condition:** --total-rows stops the selector after handling (selecting) that many rows. The stop check uses rows_handled, not rows_processed. This means it works correctly even when UPDATE affects 0 rows (e.g., records already deleted by another process or filtered out by where_clause).

## PK Source (Read PKs from File)

Read primary keys from a file instead of the database table.

**Important:** --total-rows is required when using --pk-source for accurate progress/ETA calculation.

```bash
# Count lines first
wc -l failed-ids.txt
# 1500 failed-ids.txt

# From local file (--total-rows is required)
data-updater --config config.yaml --pk-source "./failed-ids.txt" --total-rows 1500

# From local directory (processes all files)
data-updater --config config.yaml --pk-source "./failed-ids/" --total-rows 5000

# From GCS file
data-updater --config config.yaml --pk-source "gs://bucket/failed-ids.txt" --total-rows 1500

# From GCS directory
data-updater --config config.yaml --pk-source "gs://bucket/failed-ids/" --total-rows 10000
```

Or configure in YAML:
```yaml
pk_source:
  path: "gs://my-bucket/failed-ids/"
  gcs_project: "my-gcp-project"  # Required for GCS paths
  skip_header: true              # Skip first line (for BQ exports with header)
  prefetch_buffer: 5             # Number of GCS files to prefetch ahead (default: 5)
```

**GCS Authentication:**

GCS access uses Application Default Credentials (ADC). Set up with:
```bash
gcloud auth application-default login
gcloud auth application-default set-quota-project <project>
```

**File format (CSV):**
```
# Comments starting with # are ignored
12345
12346
tenant1,12345
"value,with,comma",12346
```

**Skip header (for BigQuery exports):**

BigQuery exports include a header row with column names. Use skip_header: true to skip it:
```csv
id
12345
12346
```

**Features:**
- Files are read line by line (streaming) to minimize memory usage
- GCS files are prefetched in the background to eliminate download latency (configurable buffer, default 5)
- Directory support: processes all files in sorted order
- Resume support: tracks progress per file and line number
- Can be combined with where_clause to filter PKs from file

## Status Metrics

Status logs and the status interactive command report two counters:

- **rows_processed**: rows successfully affected by the UPDATE/DELETE operation (i.e., the database reported a row change)
- **rows_handled**: rows selected and sent through the pipeline, regardless of whether the UPDATE/DELETE actually modified the row. This counter is used for progress percentage and ETA calculations

When rows_handled is higher than rows_processed, it typically means some rows were already in the desired state (e.g., already deleted or already updated by a previous run).

## Hibernate (Health-Check Based Pause)

The hibernate feature allows the processor to periodically run an external health-check script. If the script returns a non-zero exit code (indicating a problem), the processor pauses for a configurable period, then automatically resumes.

### Configuration

```yaml
processing:
  hibernate_script_path: "/path/to/check.sh"
  hibernate_pause_period: 30s
  hibernate_check_interval: 15s
```

- hibernate_script_path: Path to an executable script. The script is run at the configured check interval (default 15s). Exit code 0 means healthy; any non-zero exit code triggers hibernation.
- hibernate_pause_period: How long the processor pauses when the script signals a problem. Required when hibernate_script_path is set.
- hibernate_check_interval: How often the health-check script is executed. Defaults to 15s.

### Behavior

1. The health-check script is executed at the configured interval (default 15s) while the processor is running
2. If the script exits with code 0, processing continues normally
3. If the script exits with a non-zero code, the processor pauses for hibernate_pause_period, then automatically resumes
4. The hibernation_count metric tracks the total number of times hibernation was triggered (visible in status command output and periodic logs)

### Use Cases

- Pause when database replication lag exceeds a threshold
- Pause when disk space is low
- Pause during maintenance windows
- Any custom operator-defined health check

## Hourly Summary Log

For long-running jobs, you can enable an hourly summary log that writes JSON entries to a dedicated file. A final summary is also written on shutdown, so short-lived runs still produce a report.

```yaml
processing:
  hourly_log_path: "/var/log/data-updater/hourly.log"
```

Each JSON line includes:
- rows_processed_total / rows_processed_delta — records processed in total and during the period
- rows_failed_total / rows_failed_delta
- hibernation_count_total / hibernation_count_delta
- total_rows, rows_remaining, progress — overall progress
- interactive_commands — commands issued via socket during the period (with timestamps)
- summary_type — "hourly" or "final"

If hourly_log_path is not set, the reporter is not started.

## Auto-Interval Adjustment

Automatically adjusts the processing interval based on the hibernation ratio observed each hour. When many hibernate checks fail (high ratio), the interval increases (slows down). When the ratio is low, the interval decreases (speeds up).

```yaml
processing:
  auto_interval_enabled: true
  auto_interval_high_ratio: 0.3    # ratio >= this → slow down (default: 0.3)
  auto_interval_low_ratio: 0       # ratio <= this → speed up (default: 0)
  auto_interval_increase_factor: 1.25  # multiply interval by this to slow down (default: 1.25)
  auto_interval_decrease_factor: 0.8   # multiply interval by this to speed up (default: 0.8)
  auto_interval_min: 200ms         # floor for interval (default: initial interval)
  auto_interval_max: 30s           # ceiling for interval (default: 10x min)
```

Auto-interval can be toggled at runtime via socket commands (auto-interval on/off). See [Interactive Commands](#interactive-commands).

## Pessimistic Locking

Prevent concurrent modifications with pessimistic locking:

```yaml
processing:
  pessimistic_locking: true  # default
  lock_retry_count: 3
```

Transaction pattern:
```sql
BEGIN;
SELECT ... FOR UPDATE WHERE ID IN (...);
UPDATE ... WHERE ID IN (...);
COMMIT;
```

- MySQL 8.0+: Uses NOWAIT clause
- Sets innodb_lock_wait_timeout=1 to minimize lock wait

## Environment Variables

Use environment variables for sensitive data:

```yaml
database:
  host: "${DB_HOST}"
  user: "${DB_USER}"
  password: "${DB_PASSWORD}"
  database: "${DB_NAME}"
```

## Examples

See the examples/ directory for complete configuration files:

- minimal-config.yaml: Bare minimum configuration
- full-config.yaml: All available options with comments
- production-config.yaml: Production-ready configuration
- complex-update.yaml: Complex SQL with CASE statements
- multiline-example.yaml: Multi-line SQL using YAML block scalars
- update-sql-examples.yaml: Various update_sql patterns

## Production Tips

1. **Use environment variables** for sensitive data
2. **Enable status files** for automatic resume
3. **Set appropriate intervals** to avoid overwhelming the database
4. **Use pessimistic locking** for critical data consistency
5. **Configure replica** to offload SELECT queries from primary
6. **Test with debug mode** before running DELETE operations
7. **Use before_sql** to archive data before deletion

## Troubleshooting

### Common Issues

1. **Permission denied on socket**: Check socket path permissions
2. **Resume not working**: Verify status file path and permissions
3. **Slow processing**: Increase batch size or decrease interval
4. **Lock timeouts**: Enable pessimistic locking or increase retry count

## License

See LICENSE file in the repository root.

Turborepo Remote Cache: Accelerating CI to “Move Fast”

Tue, 17 Feb 2026 10:00:47 GMT

Hi, I’m @Zuma. I’ve been with the Web Platform Team for three months, and I’m excited to share my internship project: Turborepo remote cache.

Note: This article was originally written in March 2025. Please note that the implementation details and team names reflect the organization at that time.

Introduction

In web development, speed and efficiency are critical. A slow Continuous Integration (CI) pipeline can become a major bottleneck, hindering our ability to iterate quickly and receive feedback promptly. In essence, slow CI pipelines make it challenging to truly “Move Fast,” one of our group values. In many web repositories, the build time is a primary bottleneck that slows down the CI pipelines.

Turborepo has emerged as a powerful tool for managing monorepos, offering efficient task parallelization and caching capabilities. Ideally, Turborepo should speed up local development as well as CI pipelines. However, there is a catch in that the local Turborepo cache cannot be reused across different workflows because most CI runners including our self-hosted GitHub Actions runner are ephemeral. This limitation necessitates a caching strategy tailored to our needs, going beyond the capabilities of typical cache action which doesn’t account for task dependencies.

To overcome this challenge, we implemented Turborepo remote caching, enabling us to share a single Turborepo cache across multiple CI pipelines. This approach avoids redundant work throughout the CI workflow, significantly reducing build times and accelerating the overall CI process. While Vercel provides this functionality as a fully managed feature, its adoption is not universal. Specifically, we do not use Vercel, making self-hosted implementations of a Turborepo remote cache essential.

This blog post will go over the implementation of a Turborepo remote cache, covering the proposed architecture, performance results, future considerations, and key takeaways from the project.

Proposed Architecture

The Turborepo remote cache consists of two main components: a remote cache server and storage for saving cached artifacts. Several community implementations of remote cache servers are available, so we adopted one of them for the initial implementation.

When considering the architecture for the remote cache server, we evaluated three main approaches. The following sections detail our findings for each option.

1. Deploying a Microservice on GKE

The first approach we considered was deploying the cache server as a microservice on Google Kubernetes Engine (GKE), which aligns with our standard company practices. However, this strategy introduces significant challenges regarding latency, cost, and isolation.

Our CI cluster is located in the US, whereas our primary GKE cluster is hosted in Japan. This geographical separation results in increased latency as well as prohibitive data transfer costs of $0.08/GiB (as of this writing). After a rough cost estimation, we considered this expense to be too high, especially given the ephemeral nature of CI pods. Additionally, using a single cache server across all repositories raises concerns about cache pollution and permission management.

2. Serverless Deployment on Cloud Run

Running the cache server on Cloud Run is a popular solution in the community. Deploying Cloud Run in the US would minimize data transfer costs, and integration would be relatively straightforward with a unified TURBO_API URL.

However, each repository requires its own isolated cache to prevent cache pollution and ensure security. Similar to the GKE approach, using a single Cloud Run instance for all repositories would lead to cache pollution and complex permission management. Therefore, achieving strict separation of artifacts between repositories would require numerous Cloud Run instances, which would drastically increase computational costs.

3. Custom GitHub Action (Adopted)

Finally, we explored leveraging GitHub Actions to reduce latency and utilize existing Workload Identity on our self-hosted runners. Since our team already provides many custom GitHub Actions for web development, creating a dedicated remote cache action was a reasonable choice.

Although using GitHub Actions to run the cache server as a background job is unconventional, this approach proved to be the most cost-efficient and high-performance solution.

After comparing cost and performance, the third approach was chosen, and two custom GitHub Actions were created to provide self-service caching capability.

To support different CI workflows, we implemented two distinct patterns using custom GitHub Actions.

3-1. Background Process for Standard Builds

For standard tasks executed directly on the runner (e.g., turbo run build), we developed a custom JavaScript action. This action initializes the remote cache server as a background Node.js process.

Our custom action abstracts away this complexity. It handles the server startup and automatically configures necessary environment variables (such as TURBO_API, TURBO_TOKEN, and TURBO_TEAM) and Workload Identity.

Users only need to add a single step to their workflow:

  build:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout
      - uses: org/platform/actions/auth
+     - uses: org/web-platform/packages/turborepo-remote-cache
      - run: turbo run build

3-2. Sidecar Container for Docker Builds

For builds running inside Docker containers, accessing a process running on the runner container (as described in the previous section) is restricted due to network isolation. To solve this, we enhanced an existing custom action to launch the cache server as a sidecar container sharing the same network namespace.

We deliberately chose not to use GitHub Actions’ builtin service containers for this setup. Service containers are initialized at the start of a job, but we needed the server to start after explicitly obtaining Google Cloud credentials via Workload Identity in a preceding step.

Users can enable this feature simply by specifying an input parameter, as shown below:

  build:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout
      - uses: org/platform/actions/auth
      - uses: org/web-platform/packages/nextjs-build
        with:
          dockerfile-path: Dockerfile
+         remote-cache-enabled: true

Performance Results

The Turborepo remote cache was initially tested on our team’s monorepo. Initially, the observed improvements were minimal because our build times were already quite fast. We did achieve very high cache-hit rates due to the presence of many packages.

However, extending this to another large-scale, well-modularized repository yielded significant improvements. We achieved approximately a 50% reduction in Turbo task duration and a 30% reduction in total job duration by adjusting the CI workflow and integrating the remote cache using our custom GitHub Actions.

These figures represent the results from a workflow job building a large application on a pull request.

It’s important to note that these improvements are highly dependent on the number of applications or internal packages changed in a given commit. In fact, with a large number of changes, some cases may exhibit slower performance. This is primarily because the current remote cache server has a startup time of approximately 10 seconds. This cold start delay is particularly problematic for shorter tasks, where the startup time can negate the benefits of caching. To address this, we are considering developing a custom lightweight remote cache server to minimize startup latency and enhance efficiency, especially for shorter tasks.

Overall, despite some caveats, this resulted in a substantial reduction in the overall CI pipeline time.

On the other hand, we encountered difficulties on another repository that contains a large application lacking dependencies on internal packages. As a result, the proof-of-concept (PoC) on their pull requests did not produce an impactful outcome. However, this outcome could serve as an incentive to further modularize those repositories.

Conclusion

The Turborepo remote cache project has yielded a self-service tool that can significantly reduce CI time, enabling teams to “Move Fast”. Even with the remote cache, effective modularization remains crucial for achieving optimal speed improvements.

Through my intern project, I learned the importance of collaboration between product and platform teams. We built a remote cache solution that’s now available as a self-service tool. However, simply providing the tool isn’t sufficient. By working closely with product teams, we were able to iterate based on real user feedback.

Also, I would like to thank my mentor, azrsh, and the members of the Web Platform Team. Thanks to their feedback, especially regarding key architectural decisions, I was able to make decisions without regrets.

NavEntryScope: The missing scope in Android Hilt

Tue, 13 Jan 2026 11:00:07 GMT

Hilt, Google’s recommended dependency injection library for Modern Android Apps, proposes built-in dependency scopes that are still based on the traditional Android components hierarchy. When a Composable screen with multiple ViewModels needs to share data through common dependencies, engineers have to rely on Singletons and “manually” fix data leakage to other screens. To improve on the recommended approach, we decided to circumvent this Hilt scoping limitation by creating a new custom scope called NavEntryScope, that enables scoping any dependencies to the current navigation entry. We released the custom scope and the tools to use it as a library.

You can find the library and a working sample on GitHub.

If you want to learn more about how and when to use this library, read along.

Many ViewModels on the same screen

Hi, I’m Luca, Android Engineer on the Logistics Client team. My team is responsible for the in-app user flow taking place after an item’s checkout, from the shipping options selection to the transaction completion through peer evaluation. It’s a single screen that we internally refer to as the “Transaction screen”.

All the shipping-related steps are part of this screen. We call an API to retrieve the current transaction status upon screen opening. Based on the API response, we decide what content to display. Each Transaction status has its own Composable function and independent ViewModel to handle the user interaction. Due to the nature of Transaction Screen, it would be hard to think of one single ViewModel to handle the entire interaction, but having several ViewModels on the same screen adhering with the standard Modern Android app’s architecture is notoriously difficult.

In our example, since the initial API response’s various payloads contain data that is required by all the ViewModels active in different parts of the screen, we want to make the same response available to all ViewModels without calling the API again in each of them. We ended up implementing a Repository that exposes a data flow: the first ViewModel requests the current transaction status, while the other ViewModels observe the response flow and get notified when a new response is available.

The key requirement for this approach to work is that all the ViewModels need to share the same Repository instance in order to observe the same data flow. We initially thought that using a Singleton scope could fit our requirements, but we eventually ran into a problem. Users can have multiple Transaction screens opened in the back stack: a Singleton Repository would leak data to all the open screens in the back stack.

Our initial solution was to make the Repository return a Transaction flow mapped on their ID.

@Singleton
class TransactionRepository @Inject constructor(
   private val transactionService: TransactionService,
) {
   private val flowMap =
       mutableMapOf<TransactionId, MutableSharedFlow<Result<Transaction>>>()

   fun getTransactionFlow(transactionId: TransactionId): Flow<Result<Transaction>> =
       getOrCreateFlow(transactionId)

   suspend fun fetchTransaction(transactionId: TransactionId): Result<Transaction> =
       transactionService.getTransaction(transactionId).toDomainEntity()
           .also { result -> getOrCreateFlow(transactionId).emit(result) }

   fun cleanupFlow(transactionId: TransactionId) {
       flowMap.remove(transactionId)
   }

   private fun getOrCreateFlow(transactionId: TransactionId)
       :MutableSharedFlow<Result<Transaction>> =
       flowMap.getOrPut(transactionId) { MutableSharedFlow(replay = 1) }
}

Each screen now gets its Transaction flow through their own ViewModels, but the workaround significantly increases maintenance costs. To use this Repository, we must pass a Transaction ID to access a data flow, and remember to call the cleanupFlow() method when the main ViewModel is destroyed. With the Singleton scope bringing so much complexity, we needed to reconsider our approach.

Why a Singleton Repository?

Hilt comes with a built-in hierarchy of Dagger components and automatically handles their lifecycle.

ViewModels are assigned to a ViewModelComponent and have visibility of dependencies in the ViewModelComponent itself and ancestor components (ActivityRetainedComponent and SingletonComponent). Once we decide what component we install our module in, we can set the respective Scope annotation to retain the dependency instance until the component is dismissed.

Original image from: https://dagger.dev/hilt/components

Let’s review how each scope affects our Repository:

@Singleton makes the instance match the application’s lifecycle, which is too broad for us. The Repository will be shared by all screens of our app. That’s why we had to manually separate the data flows.
@ActivityRetainedScoped is bound to the Activity lifecycle and survives configuration changes. Since we have a Single-Activity application, this scope almost overlaps with @Singleton.
@ViewModelScoped matches the ViewModel lifecycle. Each ViewModel gets its own instance, so there’s no sharing at all.

With the Singleton annotation, our Repository is correctly shared between ViewModels, but it’s also shared with screens belonging to different navigation back stacks. Avoiding the data leakage is our responsibility.

To share data flows from other repositories, we wanted to extract the ad-hoc workaround from the Repository and achieve appropriate scoping via dependency injection. We achieved so by creating a custom Hilt component and binding it to the navigation entry lifecycle.

Let’s check how to create the custom component first.

Create the Custom Hilt Component

Hilt supports the creation of custom components and allows us to add them to its hierarchy. The steps are well documented in the library docs. Here is how we need to tell Hilt about the custom NavEntryComponent.

/** The new scope annotation */
@Scope
@Retention(AnnotationRetention.BINARY)
annotation class NavEntryScoped

/** The component that will hold our scoped dependencies */
@NavEntryScoped
@DefineComponent(parent = ActivityRetainedComponent::class)
interface NavEntryComponent

/** Builder to create component instances */
@DefineComponent.Builder
interface NavEntryComponentBuilder {
    fun build(): NavEntryComponent
}

/** Entry point to access dependencies in this component */
@EntryPoint
@InstallIn(NavEntryComponent::class)
interface NavEntryEntryPoint {
  public fun sharedRepository(): SharedRepository
}

We’ve just created NavEntryComponent as a child of ActivityRetainedComponent. However, ViewModels can’t directly access dependencies in the new component because NavEntryComponent sits alongside ViewModelComponent as a sibling component, not as an ancestor.

Original image from: https://dagger.dev/hilt/components

Hilt doesn’t allow us to make NavEntryComponent a parent of ViewModelComponent. To work around this limitation, we can build a custom bridge that gives ViewModels access to NavEntryScoped dependencies at runtime.

Access NavEntryComponent dependencies

Since our goal is to make NavEntryScope dependencies visible from a ViewModelComponent, we’ll have to provide the same dependency from ViewModelComponent too. Instead of instantiating the dependency directly, we’ll request the instance from NavEntryComponent through NavEntryEntryPoint.

Let’s see how we can create and pass the NavEntryComponent instance to a Module installed in ViewModelComponent.

NavEntryComponentStore is a simple map of screen ID and NavEntryComponent. We make it a Singleton whose instance is unique in the app and can be accessed by any Hilt component. Its responsibility is to store and return the component instance by screen ID.

@Singleton
class NavEntryComponentStore @Inject constructor() {
   private val components = mutableMapOf<String, NavEntryComponent>()

   fun storeComponent(navEntryScopeId: String, component: NavEntryComponent) {
       components[navEntryScopeId] = component
   }

   fun getComponent(navEntryScopeId: String): NavEntryComponent =
       components[navEntryScopeId] ?: error("Component not found")

   fun releaseComponent(navEntryScopeId: String) {
       components.remove(navEntryScopeId)
   }
}

We can now focus on managing the lifecycle of the NavEntryComponent instance for each screen. NavEntryComponentOwner creates both the component instance and a unique screen ID, then stores them in NavEntryComponentStore. When the screen is destroyed, the same tool will remove the component instance from NavEntryComponentStore.

The ViewModel lifecycle already matches our desired lifecycle. By making NavEntryComponentOwner a ViewModel, we can inject it via Hilt into our screen and leverage the onCleared() method for cleanup.

@HiltViewModel
class NavEntryComponentOwner @Inject constructor(
   componentBuilder: NavEntryComponentBuilder,
   private val componentStore: NavEntryComponentStore,
) : ViewModel() {

   private val navEntryScopeId = UUID.randomUUID().toString()

   init {
       // create and store component when initialized
       val component = componentBuilder.build()
       componentStore.storeComponent(navEntryScopeId, component)
   }

   fun getNavEntryScopeId(): String = navEntryScopeId

   override fun onCleared() {
       // cleanup when screen closes
       componentStore.releaseComponent(navEntryScopeId)
   }
}

We now need to pass the screen ID before injecting a ViewModel. This allows Hilt modules to retrieve the ID and obtain the current NavEntryComponent instance from NavEntryComponentStore. We do so by replacing the hiltViewModel method.

@Composable
inline fun <reified VM : ViewModel> navEntryScopedViewModel(
   vmStoreOwner: ViewModelStoreOwner = LocalViewModelStoreOwner.current,
): VM {
   val componentOwner = hiltViewModel<NavEntryComponentOwner>(vmStoreOwner)
   val navEntryScopeId = componentOwner.getNavEntryScopeId() // get screen ID
   val creationExtras = MutableCreationExtras(/* ... */).apply {
       set(DEFAULT_ARGS_KEY, Bundle(/* ... */).apply {
           // set screen ID into CreationExtras's bundle
           putString(NAV_ENTRY_SCOPE_ID, navEntryScopeId)
       })
   }

   return viewModel(
       modelClass = VM::class,
       viewModelStoreOwner = viewModelStoreOwner,
       factory = createHiltViewModelFactory(viewModelStoreOwner),
       extras = creationExtras, // provides screen ID via SavedStateHandle
   )
}

The final step is the actual “bridge” between ViewModelComponent and NavEntryComponent. We will create a new module to provide the NavEntryScoped dependencies into ViewModelComponent.

@Module
@InstallIn(ViewModelComponent::class)
object NavEntryModule {

   @Provides
   fun provideTransactionRepository(
       savedStateHandle: SavedStateHandle, // accessible in ViewModelComponent
       componentStore: NavEntryComponentStore, // Singleton
   ): TransactionRepository {
       // extract the screen ID
       val scopeId = savedStateHandle.get<String>(NAV_ENTRY_SCOPE_ID)
           ?: error("NAV_ENTRY_SCOPE_ID not found in SavedStateHandle")

       // get the stored component instance
       val component = componentStore.getComponent(scopeId)

       // obtain the entry point and return the scoped dependency
       return EntryPoints.get(component, NavEntryEntryPoint::class.java)
           .transactionRepository()
   }
}

What to change in your screen code

The most visible change is replacing the hiltViewModel() function call with navEntryScopedViewModel() for dependency injection. We have to replace it for each ViewModel that uses a NavEntryScoped dependency, directly or indirectly.

@Composable
fun UserProfile() {
   val viewModel: UserProfileViewModel = navEntryScopedViewModel()
   val state by viewModel.state.collectAsState()

   /* user profile row UI */
}

Besides that, there are two pieces of code that are not part of the library and must be updated for every new @NavEntryScoped-annotated dependency: NavEntryEntryPoint and NavEntryModule. For example, upon adding a scoped “ShippingRepository”, I need to make the following changes:

@NavEntryScoped
class ShippingRepository @Inject constructor(/* ... */)

@EntryPoint
@InstallIn(NavEntryComponent::class)
interface NavEntryEntryPoint {
  fun transactionRepository(): TransactionRepository
  fun shippingRepository(): ShippingRepository // ← newly added
}

@Module
@InstallIn(ViewModelComponent::class)
object NavEntryModule {
  @Provides
  fun provideTransactionRepository(
    savedStateHandle: SavedStateHandle,
    componentStore: NavEntryComponentStore
  ): TransactionRepository { /* bridge code */ }

  @Provides
  fun provideShippingRepository( // ← newly added
    savedStateHandle: SavedStateHandle,
    componentStore: NavEntryComponentStore
  ): ShippingRepository { /* same bridge code */ }
}

Writing this boilerplate for each scoped dependency is repetitive and error-prone. That’s why we implemented an annotation processor that automatically generates NavEntryEntryPoint and NavEntryModule, including all the scoped dependencies. All you have to do is annotate the scoped dependency with @NavEntryScoped.

Do you need NavEntryScope?

Our library makes it simple to introduce a new screen scope in your app with just a couple of code changes. However, be aware that adding a new Hilt component increases the complexity of your dependency graph in ways that may not be immediately apparent. You’ll need to make sure that your team understands how Dagger components and scopes work, and might find reduced dependency reusability across features. I suggest introducing NavEntryScope to your project if the benefits outweigh the complexity, and only scope dependencies that genuinely need to be shared within a screen.

Wrapping up

NavEntryScope bridges the gap between Hilt built-in scopes, giving us a clean way to share dependencies on the same screen. The benefit is a simpler repository deprived of the code to scope data flows and possible data leakages, and with seamless cleanup of the unused dependencies when the screen is dismissed.

While this solution continues to evolve based on feedback from teams across Mercari who’ve adopted it, I encourage you to try it and contribute.

You can find the library and source code on GitHub.

This solution was first presented (the slides are here) at droidcon Italy ‘25 (the presentation video will be made available soon).

Building EGP Cards at Merpay: Lessons from a Frontend Internship

Thu, 25 Dec 2025 10:00:01 GMT

Building EGP Cards at Merpay: Lessons from a Frontend Internship

Hello, my name is @Yusaku (Yusaku Miyata), and I’m currently interning as a Frontend Engineer on the Growth Platform team at Merpay.
This article is part of the Merpay & Mercoin Advent Calendar 2025, and I am honored to write the entry for Day 25.
I started my internship in October 2025, and I’m now in now my third month (Figure 1).
In this article, I would like to share the tasks I worked on during the internship and the key learnings I gained through hands-on development.

Figure 1: My selfies at the office

About the Team

I belong to the Growth Platform Frontend Team, which develops an internal marketing tool called Engagement Platform (EGP).
EGP enables marketers and project managers to perform CRM-related tasks—such as distributing points or coupons, creating and publishing landing pages, and managing campaigns—without writing any code (Figure 2).

Figure 2: EGP No-code Editor (EGP Content)

During this internship, I worked on improving a feature called EGP Cards.
EGP Cards allows users to create and publish card-based UI components that can be used across multiple platforms, including Web, iOS, and Android. Unlike the page editor feature (EGP Pages), EGP Cards adopts a Server Driven UI architecture, where the server returns the structure of the user interface. The content created in the editor is stored as JSON and rendered consistently across different platforms (Figure 3).
For more details on the architecture of Server Driven UI and EGP Cards, please refer to the following article by @togami and @Stefan from the same team:

Figure 3: EGP Cards Editor Screen

Task 1: Dry Run for EGP Cards

Overview: What is Dry Run?

Dry Run is a feature that allows users to simulate states by assigning mock data to variables. With this feature, users can verify content behavior before writing API calls or testing on real devices. In this task, I implemented Dry Run functionality for EGP Cards (Figure 4).

Figure 4: Implemented Dry Run Feature for EGP Cards

How It Works

The Dry Run feature works as follows:

Users enable Dry Run and input mock data into fields
The editor recursively traverses the structure tree, dynamically evaluates JavaScript code, and replaces variables with actual values
The resolved values are rendered on the canvas

Implementation Process

I proceeded with the implementation in the following steps:

Read and analyzed the existing Dry Run implementation in EGP Pages through code reading and logging
Implemented a similar feature for EGP Cards while taking Cards-specific specifications into account

During development, I searched for reusable logic between EGP Pages and EGP Cards, and carefully extracted common code into shared files to improve readability and maintainability.

Task 2: Content Agent Improvement for EGP Cards

Background: Challenges with Content Agent

EGP Content, the no-code editor in EGP, supports multiple content types such as Cards, Pages, and E-mails. Recently, an AI agent called Content Agent was introduced, enabling users to summarize or rewrite content through conversational interactions (Figure 5).
However, at the time, Content Agent did not fully understand the editor-specific constraints of each content type. As a result, it could generate content with broken UI structures, which might fail to meet users’ expectations.

Figure 5: Conversation Processing Pipeline of Content Agent

Implementation Approach

To address this issue, I implemented the following solution:

Created prompts describing the specifications and data structures of EGP Cards
Injected these prompts conditionally into the Agent Layer of Content Agent

EGP Cards has several constraints, such as not supporting media queries and requiring all elements to be structured with Flex layouts. By explicitly describing these constraints and expected outputs in the prompt, Content Agent can now generate content more suitable for EGP Cards (Figure 6).

Figure 6: Using Content Agent in EGP Cards

What I Learned

How to Deliver Output in Team Development

Through the implementation of the Dry Run feature, I learned a great deal about how to deliver effective output in a team development environment. I realized that not only the correctness of the implementation and the completeness of features, but also how pull requests (PRs) are structured and how reviews are handled, have a significant impact on the overall development efficiency and productivity of the team.
More specifically, I learned that even for bug fixes or refactoring tasks, it is important to split changes into separate PRs when the scope becomes too large or extends beyond the original task. Doing so helps reduce the review cost and makes the intent of each change clearer. I also came to understand the importance of explicitly describing the implementation intent in code and PR comments—such as why a particular approach was chosen, what alternatives were considered, and which options were intentionally not taken. This practice helps prevent misunderstandings with reviewers and leads to more constructive discussions.
When receiving reviews, I also learned that it is important not to jump straight into making fixes. Instead, taking the time to first understand the reviewer’s intent can lead to better design decisions and higher-quality implementations. In some cases, aligning on assumptions and background through discussion proved essential. Through these experiences, I strongly recognized that delivering value as a team requires not only individual coding skills, but also clear communication and a collaborative mindset.

Understanding Mercari Culture Through Real Experience

Mercari is often described as a company where transparency and flat communication give individuals a high level of ownership. Through this internship, I strongly felt this in practice.
What stood out to me the most, however, was the global, English-first development environment.
All of my previous internships were conducted in Japanese, so working in an environment where documentation, communication, and discussions were entirely in English was a refreshing experience. While I understood that smooth communication in English is essential in a global team, being able to practice this in real development work was highly rewarding.
In my daily work, I read English README files and specifications, created Pull Requests in English, and explained design decisions and concerns through discussions.
To avoid misunderstandings, I sometimes supplemented explanations in Japanese when necessary, while proactively communicating with the team.
Through this experience, I realized that Mercari’s culture is not just a slogan, but something deeply embedded in everyday work.

Rediscovering the Depth of Technical Challenges

Until now, I had mainly worked in frontend development through internships and personal projects, and I felt that my learning in this area might be approaching a plateau.
However, working on EGP completely changed that perception. While EGP provides a highly interactive and rich UI, it is supported by complex internal logic, such as no-code content creation and delivery mechanisms, as well as safe and efficient interactions with AI agents.
During the tasks, I received requirements at a relatively abstract level and broke them down into concrete implementation steps on my own. While learning how EGP is used, I also proposed improvements—such as adding image preview functionality—that could enhance user experience.
In addition, when improving Content Agent, I designed the implementation so that it would not be limited to Cards only, but could be easily extended to other content types like Pages and E-mails in the future. By separating prompts by content type, I focused on readability and extensibility.
This experience taught me that designing with a long-term product perspective directly contributes to better user experience, improved efficiency, and ultimately business value—something I find particularly compelling about Mercari.

Closing

Through this internship, I was able to learn not only technical skills, but also how to approach engineering from the perspective of product value and team collaboration.
I hope to continue applying these learnings to my future development work and personal growth as an engineer.
Thank you very much for reading.

When Speed Wasn’t About Coding Faster: Our Journey to ‘One Person One Release’

Tue, 23 Dec 2025 12:00:53 GMT

This post is for Day 23 of the Mercari Advent Calendar 2025.

Introduction

Hi, I’m Jieqiong Yu, an Engineering Manager working with the Shops & Ads Mobile Enabling team at Mercari.

Over the past six months, my team in Shops & Ads Mobile Enabling has been supporting cross-platform feature development on iOS and Android. On paper, we had everything right: strong engineers, solid foundations, and a clear roadmap. Yet, we started noticing something subtle: Delivery felt slow.

Not "heavy" in terms of coding speed—our engineers could generate code quickly enough—but heavy in the coordination required before a single line of code is written.

This is the story of our experiment with the ‘One Person, One Release’ philosophy – why we tried it, how it worked in practice on iOS and Android, and what it taught us about ownership, coordination, and growing engineering capability with AI as an enabler.

The Hidden "Coordination Tax"

At some point, we realized something uncomfortable: the bottleneck wasn’t implementation speed anymore.

Engineers across iOS, Android, Web, and Backend were fully capable of building complex features within their own domains. When work was clearly defined, teams moved fast.

The friction lived in the "in-between."

Specifications arrived incomplete and needed to be shaped through discussion. Even with a shared Figma design, multiple engineers across platforms had to align on the same problem statement, clarify expected behavior, and agree on edge cases. API contracts became a frequent coordination point. Defining request and response structures, naming fields, and agreeing on domain vocabulary required repeated conversations across mobile, web, and backend. Each platform brought its own conventions, and aligning those conventions took time.

By the time implementation began, a significant amount of energy had already been spent just reaching a shared understanding of what we were building and how the pieces fit together.

We still shipped. But delivery felt heavier every cycle.

What we were losing wasn’t engineering capability – it was shared understanding, clear ownership, and the space for engineers to move fast and grow beyond a single platform without paying constant coordination costs.

‘One Person One Release’ philosophy emerged as an experiment to restore our momentum. We asked ourselves a fundamental question:

Can a single engineer lead a feature from design to delivery across various tech stacks (iOS, Android, Web and backend)?

Delivery slows down – not because engineers move slower, but because too many people need to move together.

Starting Small: The First Experiment

Our first opportunity came when we began migrating one of our web view screens to native code on iOS and Android (Shops Item Detail migration), but we deliberately started small. Instead of tackling a large migration or a complex surface, we chose a contained user story: showing the last purchased date on the item detail page. The scope was simple enough to experiment safely, yet real enough to reflect how we build production features.

Rather than splitting the work by platform – Android here, iOS there – we asked a single engineer to deliver the feature end-to-end across iOS and Android.

We didn’t change the definition of done. We didn’t relax code review standards.

What we changed was how the work was owned.

The engineer started on the platform they were most familiar with. Before moving to the other platform, they spent time learning the basics: understanding the code architecture, setting up the environment, figuring out how to build, test, and debug. This wasn’t something an AI agent could do magically on its own.

To make that learning curve manageable, we leaned heavily on pair programming sessions. Engineers walked each other through platform-specific patterns, common pitfalls, and project conventions. This human knowledge transfer was essential.

Once that foundation was in place, AI agents became a powerful enabler.

Engineers used AI agents to help translate pull requests from one platform to the other, generate boilerplate code, and surface relevant platform APIs. Instead of starting from a blank file, they could focus on validating behavior, adapting logic idiomatically, and ensuring quality. Reviewers stepped in where deeper platform expertise was needed – not to take over, but to guide.

The result was eye-opening.

The feature shipped faster than expected, with fewer inconsistencies and far less back-and-forth alignment. More importantly, the engineers gained confidence that they could deliver beyond their primary platform – without sacrificing quality.

That small win gave us the signal we needed. We realized it was a workflow worth scaling.

Turning an Experiment into a Habit

As we applied new ways of working to more features, patterns started to emerge – not as principles on a slide, but as friction we could feel day to day.

The first thing we noticed was how much the experience depended on the foundation underneath. When core pieces like networking, navigation, or analytics behaved similarly on iOS and Android, engineers could move between codebases with confidence. When they didn’t, progress slowed immediately. Even small inconsistencies forced engineers to stop and re-orient themselves, breaking the flow that was meant to be created?

Naming conventions proved far more critical than we anticipated. Over time, independent naming patterns for screens, data models, and component boundaries had drifted apart, creating significant cognitive load. When a single engineer was responsible for developing on both platforms, those differences surfaced instantly. Aligning conventions didn’t just make the code easier to read – it made it easier to think about the system as a whole, and it made AI-assisted translation far more effective.

The role of code reviewers shifted fundamentally to support the ‘One Person One Release’ philosophy. Platform specialists moved away from being final gatekeepers to becoming early-stage guides. The most effective reviews focused on identifying platform-specific nuances and sharing best practices. This helped engineers course-correct before small issues became structural ones. That shift required trust on both sides, but it paid off quickly.

AI played a critical role – but not in the way we first imagined. It didn’t magically produce correct cross-platform implementations. Engineers still had to invest time learning the basics of the other platform: how the code was architectured, how the files were structured, how to build and test, how state flowed through the app. Pair programming sessions were essential here. Once that understanding was in place, AI became a real accelerator – translating pull requests, generating boilerplate, and reducing the cost of repetitive work – while engineers remained firmly in control of correctness and quality.

Over time, ‘One Person One Release’ philosophy stopped feeling like an experiment we were “trying out”. It became a lens that exposed where our systems were easy to work with – and where they weren’t. And that, more than speed alone, turned out to be its real value.

Applying ‘One Person One Release’ philosophy to Real, Complex Features

The earlier experiments taught us an important lesson: the approach of implementing on one platform and then using AI agents to translate that work to the other platform was effective – but only within clear limits.

For small, contained user stories, it worked surprisingly well. An engineer would build on the platform they knew best, and the AI agent could help carry that logic across to the other platform. With stable foundations, consistent conventions, and careful reviews, we could move fast without drifting.

Those limits became obvious when we tried something bigger.

When we moved to more complex work – for example, implementing the coupon features on the Shops item detail page – the using the AI to translate PR from one platform to another platform approach started to fail. The scope was wider, dependencies were heavier, and the behavior had more edge cases. Translating after the fact became noisy: the generated code needed too much correction, and the feedback loop got slower instead of faster.

That pushed us to try a second approach.

By then, engineers had already spent enough time working across platforms that they weren’t just “visiting” the other codebase anymore. They could navigate it, understand low-level implementations, and reason about platform-specific trade-offs. That gave us a new foundation to build on.

Instead of translating a PR from one platform to another, we started generating both iOS and Android code from the same prompt. We built a set of cross-platform prompts that embedded what we had learned: our architecture choices, best practices, and constraints for each platform. Engineers would feed in the same spec, generate both implementations, and then debug and refine them directly on each platform until they were correct and shippable.

In practice, this felt very different. The “source of truth” stopped being a PR on one platform. The source of truth became the shared spec plus the shared prompt structure – and engineers validated the output by running, testing, and reviewing it on both iOS and Android.

What We Learned About Engineering Through ‘One Person One Release’

‘One Person One Release’ philosophy wasn’t only about speed. It taught us lessons about architecture, quality and engineering culture.

It exposed the cracks in our foundation – When a single engineer drives features across all platforms, inconsistencies surface immediately. We uncovered areas where naming conventions drifted, common patterns diverged, and design mismatches forced unnecessary rework. Fixing these issues didn’t just help the immediate release—it hardened the entire codebase.
It cultivated systems thinking – Working across boundaries forced engineers to broaden their perspective. This led to richer design discussions and a better ability to anticipate the downstream consequences of technical decisions.
It proved that AI demands structure – We learned that AI-native engineering thrives on predictability. Without consistent architecture and naming, AI tools generate noise. But with strong guardrails, they transform from simple assistants into true force multipliers.

And finally, ‘One Person One Release’ philosophy clarified that engineering velocity isn’t only about “writing code faster” – it’s about reducing friction in the entire development loop.

Moving Forward

We’re still learning.

‘One Person One Release’ philosophy is not a magic solution, and it’s not something we expect to apply everywhere. There are still areas – especially deeply platform-specific surfaces – where starting with platform experts is simply the right choice. The philosophy acknowledges this constraint rather than attempting to replace it.

What it has given us, though, is another option.

We’ve found it particularly effective in situations where specifications are evolving quickly, where teams need fast feedback to make progress, or where a feature spans multiple platforms with largely shared structure. In those moments, reducing coordination overhead and clarifying ownership early makes a noticeable difference.

As we continue building features and strengthening our engineering foundations, ‘One Person One Release’ philosophy has become one of the tools we reach for when we need both speed and consistency. It pushes us to think more holistically about the system, to design architecture that’s easier to move across, and to treat AI not as a shortcut, but as a broader development model that still depends on solid engineering judgment.

Looking back, this initiative has been one of the most meaningful engineering experiences for me this half year – not because it changed how we write code, but because it changed how we think about building together.

We’re still exploring. Still experimenting. Still refining what works and what doesn’t. And that feels exactly right for where we are headed next!

Tomorrow’s article will be by @kiko and @aisaka. Look forward to it!

Tales of OIDC & OAuth Security: What It Takes to Trust a Token

Mon, 22 Dec 2025 11:00:10 GMT

This post is for Day 22 of Mercari Advent Calendar 2025, brought to you by @Kahla from the Mercari Product Security team.

In this article, we will explore OIDC and OAuth flows, examine common related attacks, and discuss practical hardening strategies.

Background

Recently, in an initiative to improve the security of our OIDC server and identity flows, the product security team collaborated with the identity provider (IDP) team on threat modeling and knowledge-sharing sessions. As I was the most involved in this project, I had the opportunity to deepen my knowledge of the different IDP flows. I thought this article could be a great opportunity to share our takeaways and a security testing guide for similar systems.

Overview of OIDC and OAuth2 flows

Before diving deeper into the security aspects, let’s start with a quick reminder about OAuth2.0 and OIDC. Historically, the OAuth protocol was first introduced as an industry-standard authorization protocol, allowing third-party applications to access a user’s resources with limited permissions without requiring direct access to the user’s credentials. As OAuth was mainly meant for authorization, OIDC came to fill in the gap and build the identity layer on top of OAuth2.0.

OIDC introduced the ID token, which is a JWT containing identity claims used to identify the user. There is a common misconception that OAuth can be used for authentication. Many applications attempt to work around this limitation, but these approaches often introduce design flaws and security issues. The root cause is the nature of the access token: it isn’t standardized and is intended solely for accessing protected resources, not for identifying users.

To make things clear, below is a sequence diagram for a regular OIDC flow:

In Step 3, the client redirects the user’s browser to the authorization server’s /authorize endpoint with the required parameters (such as client_id, redirect_uri, response_type=code, scope, state, nonce, etc.). The user authenticates on the OIDC/authorization server side.
After successful authentication (Step 4), the authorization server issues a short‑lived authorization code and returns it to the client via the callback endpoint, which must match the redirect_uri that was sent in the initial request.
The client backend then exchanges this authorization code at the token endpoint for tokens: in OIDC, an id_token and an access_token (and possibly a refresh_token); in pure OAuth 2.0, only an access_token is returned.

Compared to a plain OAuth 2.0 flow, the main differences in OIDC are the presence of the id_token (for authentication) and the use of OIDC-specific scopes such as openid and profile in the scope parameter.

OIDC Flow Security

In this section we will go over the most important security related components in OIDC flows and discuss the related attacks and mitigations.

State Parameter

The state parameter was introduced to protect against Cross-Site Request Forgery (CSRF) attacks. It is generated before redirecting the user to the /authorize endpoint and stored on the client side. When the user returns to the callback page, the received state value is compared with the originally stored one to ensure the request is legitimate.

If the state parameter is absent or lacks proper verification, an attacker can lure the user into visiting a callback page with the attacker’s code and get them logged in to the attacker’s account. In some cases, an attacker may even be able to carry out more severe CSRF attacks.

It’s also common to see additional information encoded into the state parameter, which may represent a risk when it’s not verified correctly and an attacker can modify it.

The responsibility of securely generating and verifying the state parameter falls on the client application.

Redirection Behavior (redirect_uri)

The redirect_uri parameter is used by the OIDC server to redirect the user back to the client application’s callback endpoint. This parameter should be strictly verified against a pre-configured whitelist. It’s preferred to have strict comparison here for both the application’s domain and endpoint. The main reason is to avoid common URL comparison pitfalls such as:

Relying on endsWith() or startsWith() style logic: This is usually bypassable by registering domains similar to attacker-mercari.com or mercari.com.attacker.com.
Loose comparison of the path part: A common mistake is to assume a redirect is safe as long as the path begins with the expected prefix on the same domain (e.g., allowing anything that starts with /callback). This becomes dangerous when the application contains an open redirect somewhere else on the site.

Example:
Suppose the authorization server validates that the redirect_uri starts with https://example.com/ and therefore accepts:
https://example.com/shop?next=https://attacker.com
If /shop contains an open redirect via the next parameter, the authorization response is first sent to a legitimate endpoint on example.com, but then immediately redirected to https://attacker.com.

In OIDC Hybrid Flow (e.g., response_type=code id_token), the authorization server returns some tokens directly in the URL fragment (#id_token=…). Fragments are handled entirely in the browser and survive redirects, even across open redirect chains.
As a result, if your redirect_uri validation can be bypassed through an open redirect, the ID token included in the fragment can be carried all the way to the attacker’s domain, leaking it without ever touching your own callback handler.

The following diagram explains this attack scenario:

These are only basic examples; multiple other bypasses are available. The PortSwigger article on URL validation bypasses can be a good reference The correct validation of redirect_uri is the authorization server responsibility.

Nonce Value

Nonces (“numbers used once”) are specific to OIDC flows and are designed to protect against replay attacks in which an attacker attempts to reuse a previously issued ID token. When initiating the authorization request, the client generates a cryptographically random nonce value and stores it.

During the callback, the client must verify that the nonce claim inside the returned ID token exactly matches the stored value. This ensures that the token was generated specifically in response to this authorization request and cannot be replayed from another session or user.

Importantly, nonces must be single-use: once a nonce has been validated, it should be discarded so it cannot be matched again in a future flow. If nonce validation is missing, weak, or allows reuse, an attacker can replay an ID token issued in a different context, effectively “recycling” a past authentication and impersonating the original user.

Proof Key for Code Exchange (PKCE)

The Proof Key for Code Exchange (PKCE) flow is primarily designed to mitigate authorization code interception. These attacks are particularly common in mobile applications for example, in cases of deep-link hijacking. PKCE introduces an extra layer of defense through a code verification mechanism.
The client app begins by generating a cryptographically random code_verifier. It then derives a hashed value from it, known as the code_challenge, and sends this challenge to the OIDC server during the initial authorization request.
Later, during the token exchange phase, the client must provide the original code_verifier. The OIDC server re-computes the hash and compares it to the previously received code_challenge. If they match, the server knows that the party performing the exchange is the same one that initiated the flow. This ensures that even if an attacker intercepts the authorization code, they still cannot exchange it for tokens because they do not possess the original code_verifier.
It’s interesting to note that even though PKCE prevents code interception in most cases, if the attacker manages to trigger the OIDC flow using their own controlled link (getting the user to click or be redirected to their URL), the attacker will be able to use the intercepted code successfully, as the initial code_challenge value was attacker-controlled. However, such an attack is quite hard to apply in real life given the multiple requirements.

The blue-highlighted sections represent the PKCE-specific components of the flow: the creation and transmission of the code_verifier and code_challenge, and later, the server-side verification of the original code_verifier during the token exchange.

Demonstrating Proof of Possession (DPoP) token

Demonstrating Proof of Possession (DPoP) is mainly used to protect against token theft. It’s a JWT sent along with every request to prove possession of the access token. This is cryptographically ensured by the fact that the app first generates a key pair where the public key will be shared with the authorization server and the private key will be used to sign the DPoP JWT.

The public key will be used to verify the DPoP token for every request, proving possession of the token. One crucial step is to bind the access token with the DPoP public key when it’s first issued.

A lot of implementations ignore this step, making the presence of DPoP useless, as the attacker can just forge a DPoP token using their own key pair and reuse the stolen access token depending on the scenario.

iss Parameter

The iss (issuer) parameter is usually returned by the authorization server in order for the client to confirm the expected authorization server. This is mainly introduced to prevent mix-up attacks, which happen when the client can’t determine which authorization server to use to exchange the code value when multiple authorization servers are implemented (e.g. “Sign in with Google”, LINE, etc. on the same website).

Such attacks aim to leak the code value to a malicious attacker-controlled authorization server. They are quite common within applications involving multiple users or organizations sharing the same callback endpoint while allowing registration of a custom identity provider (for example, a SaaS product giving organizations the option to enable custom SSO).

Exploiting this issue differs by case; however, it’s often related to understanding how the application logic decides which OIDC server to use when exchanging the code value. A great description of an attack example is described in the RFC here.

The following sequence diagram illustrates the overall attack idea. The ways to confuse the client still depend on the situation and implementation.

To mitigate mix-up attacks, the client must ensure that authorization responses are bound to the correct authorization server. This typically involves validating the issuer (iss) value returned in the response and rejecting any mismatch. Using distinct redirect URIs per provider and relying on trusted metadata further reduces the risk of confusing authorization servers.

Hardening of OIDC/OAuth Flows

Understanding the previous attacks, why every parameter exists, and making sure to implement them in the correct way will already help mitigate most of the common issues. If you are seeking to protect a highly sensitive API, then FAPI 2.0 security profiles might be a good resource to check. They define security improvements and mitigations based on the following attacker model, covering protections even at the network layer and recommendations per component.

One interesting hardening extension is adopting Pushed Authorization Requests (PAR). Instead of navigating directly to the /authorize endpoint with all the needed parameters, a POST request is first sent with these parameters to the authorization server. The server then returns a request_uri that will be used afterward in the /authorize request.
This moves sensitive request details from the public browser (front channel) to a secure server-to-server (back channel), preventing exposure, tampering, and URL length issues. This is only a general idea about PAR, as it also involves other checks and requirements that need to be satisfied. More details are in the official RFC page.

An overall diagram is presented below for the PAR extension:

Summary

We discussed the main attack vectors that can affect OAuth2.0/OIDC flows and how each parameter and component can help mitigate them. However, to keep the article concise, some in-depth details were intentionally left out.

At their core, OIDC and OAuth security mechanisms revolve around establishing and preserving three key assurances:

Is the user (resource owner) really the legitimate user?
This addresses replay protection, token theft, DPoP, nonces, and every mechanism that ensures tokens cannot be reused or repurposed by attackers.
Did the user intentionally perform the action?
Answering this mitigates CSRF and prevents malicious sites from initiating or influencing an OIDC/OAuth flow without the user’s awareness.
Is the client application truly the one that should receive the tokens?
This includes validating redirect URIs, enforcing PKCE correctly, and ensuring state/nonce correlation.

All of these assurances rely on one more important consideration:

Is the environment itself secure from client-side vulnerabilities (like XSS) that could undermine the entire flow?

If any of these assurances fail, the OAuth2.0/OIDC flow becomes susceptible to compromise. When all are satisfied, the system maintains strong guarantees about identity, intent, and trust between the user, the client, and the authorization server.

After working on this initiative, it has become clear to me that having a clear threat model before beginning any assessment is really important, especially when dealing with complex flows. It helps ensure no attack vector is missed and is also a great way to learn more from the owner team. Big kudos to the @IDP team members for the amazing collaboration.

If you are interested in learning and working on similar fun projects, feel free to check our careers page for job openings!

Tomorrow’s article will be by @Sneha & @Yu. Look forward to it!

Nine Months of DevEx Improvement at Mercari Group

Sun, 21 Dec 2025 10:00:05 GMT

Introduction

This post is for Day 21 of the Merpay & Mercoin Advent Calendar 2025\.

Hi, I’m ntk1000, an Engineering Manager for the KYC and Partner Platform teams at Merpay. Six months ago, we introduced our company-wide initiative to improve Developer Experience (DevEx) across Mercari Group. We designed a quarterly improvement cycle, achieved 100% participation from engineers and EMs, and identified structural challenges in areas like Deep Work (uninterrupted time for focus) and cross-team Collaboration.

During the first six months (FY25 Q4 → FY26 Q1), our overall developer experience metrics showed little change. While some teams demonstrated significant improvements, many remained stagnant. After reassessing our approach based on this reality, the most recent quarter (FY26 Q2) saw substantial improvements across the organization, particularly in Deep Work. We achieved improvement levels typically considered annual targets within a single quarter, clearly demonstrating the effectiveness of our approach shift.

This article shares our nine-month journey, achievements, and efforts to scale DevEx improvements across the organization.

Scaling Improvements: From Teams to Organization-Wide

Phase 1 (FY25 Q4 → FY26 Q1): Localized Success and Overall Stagnation

The FY26 Q1 survey showed most organizations’ developer experience metrics remained flat. However, analyzing specific challenge areas and team-level data revealed successful improvement cases.

Analysis of Success Cases

For Deep Work specifically, we investigated teams that achieved significant improvements and found common patterns:

Established clear policies to protect focus time
- Organized meetings, defined and implemented consolidation rules
Set team-wide "No Meeting Time" blocks
- Regularly held engineer focus days—e.g., monthly two-day periods completely free of meetings
Automated routine tasks using AI tools

The common thread in these initiatives is policy-based improvements supported by team-wide commitment. Rather than individual efforts, they were implemented as structural changes agreed upon by entire teams, with policy-based approaches enabling rapid deployment and adoption.

However, team-level implementation resulted in inconsistent improvement responses and prevented natural horizontal deployment of success patterns.

Phase 2 (FY26 Q1 → FY26 Q2): Fusion of Bottom-up and Top-down

Recognizing the limitations of team-level improvements alone, we built a system to simultaneously activate both bottom-up and top-down approaches. The two-layered improvement cycle can be summarized as follows:

Bottom-up:

Team-level improvement cycle established from FY25 Q4:

Actors: Individual Contributors (ICs) and Engineering Managers (EMs)
Characteristics:
- Agile and responsive to team-specific challenges
- High success rate in areas teams can control and respond to quickly (Deep Work, Documentation, Code Maintainability, etc.)

Top-down:

Organization-level improvement cycle fully established from FY26 Q2:

Actors: VPoE, Directors, Managers of Managers (MoMs)
Purpose:
- Address structural challenges difficult for individual teams to solve
- Cross-organizational policy decisions and resource allocation
- Standardization and horizontal deployment of success patterns

Specific Implementation Example

Within the Fintech organization (which includes Merpay/Mercoin), requests for Deep Work improvement remained consistently high while scores remained low. Under VPoE leadership, Deep Work improvement was designated as an organizational priority, with the following initiatives:

Visualization of Deep Work-related metrics across Fintech organizations (Deep Work scores, proportion of meeting-heavy days and interruptions)
Horizontal deployment and localization of Deep Work improvement examples
- Review and consolidation of regular meetings within the organization
- Sharing and standardization of success cases across teams
More detailed challenge analysis using additional surveys
Support for EMs in executing improvements
Progress tracking
Reporting to executive leadership, sharing initiatives with non-engineering organizations

Organization-wide averages that hadn’t changed through individual team efforts alone achieved approximately 16% improvement in a single quarter by combining VPoE-led top-down initiatives with EM/IC-led bottom-up execution. These results have been reported to executive meetings and are scheduled to be shared at company-wide gatherings.

The clear policies protecting focus time that proved effective for Deep Work improvement are considered applicable to non-engineering organizations as well. We expect this will lead to company-wide Deep Work improvements.

Additionally, since this round primarily involved policy-based improvements with rapid deployment and adoption, effects were more easily realized across the organization. Going forward, we need to address not only easily tackled short-term initiatives but also more challenging issues requiring long-term responses.

Learnings from DX

In redesigning our approach, insights from a presentation by DX (the company providing our DevEx platform) proved valuable. We’d like to briefly share these learnings.

Initiatives Need Structure

The presentation emphasized that many DevEx improvements fail not due to lack of team interest, but due to lack of proper structure. Three components were identified as essential for success:

1: Build the Business Case

Translate developer pain into business outcomes, not just explain the pain:

Connect to organizational KPIs: Time loss, cost, quality, retention
Quantify impact at scale: 20 minutes lost per build per developer × 700 engineers \= significant cost
Show value unlocked: Not just removing pain, but what becomes possible (faster features, higher reliability, better recovery)

2: Structure the Initiative Properly

Timebox for 6-12 months minimum: Real change takes time. Sprints produce quick wins but don’t form habits.
Set natural checkpoints: Baseline → midpoint → end measurement
Enable team mobilization: Give time for communication, planning, and embedding practices

3: Define Appropriate Metrics

Northstar KPI: Productivity, satisfaction, quality, retention
Leading indicators: What teams can directly influence (e.g., build time, review time)
Guardrails: Prevent gaming (e.g., PR count alone can be artificially inflated)

Three Essential Roles

All successful initiatives require:

Executive Sponsor: Provides top-down leadership and business alignment (e.g., VPoE decision-making)
Champion: Frames problems with data and provides tactical guidance (e.g., Directors and MoMs understanding organizational data and determining direction)
Manager: Allocates time and translates initiatives into team-level actions (e.g., EMs and ICs improving teams)

Without any of these roles, initiatives stagnate.

Sustainability Through Visibility

To prevent initiatives from stalling midway, the following mechanisms are recommended:

Visualization: Dashboards showing progress/lack of progress
Operational reviews: Regular meetings with challenges/actions/outcomes
Clear leadership: Accountability for progress, business alignment

Ongoing Challenges

Some challenges remain unresolved, and we continue working on them.

Difficulty of Horizontal Deployment

Success patterns don’t spread automatically. Mercari Group particularly has diverse product phases and organization sizes, so approaches for mature large organizations may not apply to small organizations in startup phases.

Current efforts: Rather than simply collecting success cases, we’re organizing knowledge by including information like organization size and product phase, enabling each organization to autonomously identify applicable actions.

Measuring Long-term Impact

While we can measure survey scores and immediate metrics (meeting time, interruption frequency), connecting DevEx improvements to business outcomes (delivery speed, quality, retention) remains difficult.

Current efforts: We’re analyzing correlations between DX score improvements, various survey results, and quantitative data. No definitive conclusions yet.

Multi-quarter Continuity

While high participation rates continue, we’re seeing signs of survey fatigue and frustration with issues not improving.

Current efforts: We continue adjusting DX surveys to prevent bloat and holding internal DX-related events to regularly communicate significance.

Summary

Nine months ago when we started this initiative, we achieved strong engagement, identified structural challenges, and created action plans.

Currently, we’re at the stage of transforming initial momentum into sustained organizational change. We’ve moved from a situation with wide variance—some teams achieving significant results while others faced challenges—to being able to drive cross-organizational improvements.

Elements needed to scale DevEx improvement organization-wide:

Structural support (improvement systems, role clarity)
Cultural commitment (leadership from multiple directions, regular visibility)
Practical frameworks (deployment of success cases teams can adapt)

We must continue to remember that improvement scores are indicators, not objectives. More importantly, we must maximize business outcomes by ensuring teams acquire these capabilities:

Identify friction in daily work
Clearly express issues to leadership
Take collective action for improvement
Measure and reflect on results

DevEx as Continuous Practice

Continuous practice is essential to quickly identify and address challenges as products evolve, organization size and structure changes, and AI transforms development practices.

The goal is not achieving perfect scores, but building organizational capability to continuously sense and respond to developer experience issues, thereby increasing productivity.

While Phase 1 saw flat organization-wide overall developer experience metrics, Phase 2 achieved significant improvements. We’ll continue improving this initiative itself to ensure this change becomes an ongoing improvement process rather than temporary.

Tomorrow’s article will be by @taki. Look forward to it!

Capturing Network Packets in Kubernetes

Thu, 18 Dec 2025 11:00:28 GMT

This post is for Day 18 of Mercari Advent Calendar 2025, brought to you by @mshibuya from the Mercari Platform Network and SRE team.

Today, I’m going to talk about capturing network packets in a Kubernetes environment. As mentioned above, I’m currently part of the Network team, where we build and operate the network-related components among the various platform components that support product development at Mercari.

At Mercari, we operate over several hundred microservices, and the network communication both within and between these services is complex and diverse. Due to the nature of our work, the Network team is often asked to investigate network-related issues and problems that arise in this environment. Of course, sometimes the cause is a simple misconfiguration, but in situations where the problem is complex and we’re struggling to find a starting point, we need a means for deep analysis. This is where packet capturing comes in.

For this kind of investigation, if the execution procedure itself is not clearly defined, it won’t be useful when a problem actually occurs—especially if it’s a high-urgency incident. The method we’ve established might not be directly applicable to your environment as-is. However, the purpose of publishing this article is my belief that introducing a stable and executable investigation procedure will be helpful for all of you when you create similar procedures in your own organizations.

Why is Capturing Packets in Kubernetes Difficult?

Kubernetes provides abstractions at various layers, such as hardware and OS, offering an environment where developers can run workloads without being bothered by such raw resources. For security reasons, users like developers generally do not have access to the raw nodes. Furthermore, workloads like Pods running on them are isolated from each other in a multi-tenant fashion. Therefore, it’s not as simple as the old days where you could just run tcpdump on a server and call it a day.

There’s also the difficulty due to the Service Mesh. At Mercari, we have adopted Istio, and communication within the cluster is basically encrypted by mTLS. This means you can’t see the content of the communication as it is. We needed to establish a method that takes this into account.

Furthermore, our standpoint as a Platform team is to provide a complete set of tools for developers to deliver features easily and quickly, including these Kubernetes clusters. It’s impossible to predict when the need for such deep network troubleshooting will arise. A crucial requirement was to enable developers to perform this kind of investigation themselves via self-service, without needing special Platform-specific privileges.

Pod-Level Capture Using Ephemeral Containers

The method we established to meet these conditions is one that utilizes Kubernetes’ Ephemeral Containers feature.

Ephemeral Containers became generally available (GA) in Kubernetes v1.25. They allow you to attach a temporary debugging container to a running Pod’s shared resources, such as its network namespace, without needing access to the entire node. This is perfect for packet capturing, as it eliminates the need to include debugging tools like tcpdump within the application container. Another significant advantage is that it doesn’t require special privileges for the Node or the entire Cluster, allowing both Platform members and developers to conduct investigations using the same method.

The specific procedure is as follows.

Step 1. Getting Necessary Permissions

At Mercari, we use an in-house tool called Carrier to temporarily grant permissions, achieving Zero Touch Production where we normally do not have operational privileges in the production environment.
Therefore, when performing packet captures to investigate problems in production, we first need to obtain a Role that has operational permissions for the target Pod.

This Role is pre-configured with the necessary permissions to operate Ephemeral Containers.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: example-role
rules:
# ...
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["create", "delete", "deletecollection", "patch", "update"]

Step 2. Launching the Ephemeral Container

Once you have the permissions, attach an Ephemeral Container to the target Pod. Here, we use netshoot, which comes with a rich set of tools for all kinds of network troubleshooting, including packet capturing.

kubectl debug -it -n <your-namespace> <target-pod> \
  --image=nicolaka/netshoot \
  --custom=./root.yaml --container=netshoot-debug

Here, we prepare a file ./root.yaml with the following content beforehand.

securityContext:
  runAsUser: 0
  runAsNonRoot: false

This fulfills the requirement of "running the netshoot container as root," which is necessary to execute tcpdump inside the container. It’s not a very long piece of content, so I’d love to write it inline in the command, but for now, it seems that kubectl debug can only take a file as an argument…

Step 3. Performing the Capture

Once the netshoot container’s shell opens successfully, you can start the capture. Here, we’re writing to the file /tmp/capture.pcap.

tcpdump -i any -w /tmp/capture.pcap

In an Istio-enabled environment, this -i any is the key point. Traffic passes not only through eth0 but also through virtual interfaces redirected by iptables. To avoid missing any of this, we target all interfaces. If you only capture on eth0, you’ll likely only get the mTLS-encrypted content, which should be insufficient for investigation purposes.

Capturing all traffic can result in a massive amount of data. I won’t go into details here, but you can filter the packets you capture using tcpdump options. It’s easier for later analysis if you narrow down the capture as much as possible to packets related to the problem you’re investigating. Of course, there’s a trade-off: if you filter too much, you might find that you "didn’t capture the necessary data."

Step 4. Retrieving the File

The above step creates a file in the Ephemeral Container. You can then download it from your local machine using kubectl cp to complete the process. Don’t forget to specify the container name you assigned in Step 2.
Now you can move on to analyzing the captured data.

kubectl cp -n <your-namespace> <target-pod>:/tmp/capture.pcap ./capture.pcap -c netshoot-debug

Once you get comfortable with the process, you might want to perform Steps 2-4 in a single line. It would look like this. You use -iq to prevent extraneous output from being mixed into the file and also discard the standard error output. The -G 10 option specifies the capture duration in seconds.

kubectl -n <your-namespace> debug <target-pod> -iq --image=nicolaka/netshoot --custom=./root.yaml -- bash -c 'tcpdump -i any -G 10 -W 1 -s0 -w - 2>/dev/null' > tcpdump.pcap

Node-Level Capture

In addition to the Pod-level capture method above, we have also prepared a procedure for performing packet captures by SSH-ing into a Google Kubernetes Engine (GKE) Node and using the CoreOS Toolbox. However, this is considered a supplementary method because it requires privileges to SSH into the Node and, as mentioned earlier, it can only capture the encrypted Istio traffic. It is mainly intended for Platform members to use for troubleshooting issues that can only be observed at the node level.

Step 1. Getting Necessary Permissions

At Mercari, we build and operate our Kubernetes clusters with Google Kubernetes Engine. First, you need to obtain the necessary permissions to SSH into the GKE nodes using the aforementioned Carrier. The following permissions should be sufficient.

roles/compute.instanceAdmin.v1
roles/iam.serviceAccountUser
roles/iap.tunnelResourceAccessor

Step 2. Identifying the Node

Use the kubectl get pod command to check the name of the node where the target Pod is hosted.

$ kubectl get pod -n <your-namespace> your-app-pod-7f5b7f7d9f-abcde -o wide
NAME                           READY   STATUS    RESTARTS   AGE    IP           NODE                                NOMINATED NODE   READINESS GATES
your-app-pod-7f5b7f7d9f-abcde   2/2     Running   0          2d1h   10.1.2.3     gke-cluster-1-node-pool-1-a1b2c3d4   <none>           <none>

Step 3. Entering the Toolbox Environment

Use gcloud compute ssh to SSH into the node, and then use the toolbox command to enter a shell environment equipped with debugging tools.

gcloud compute ssh --project <your-project> gke-cluster-1-node-pool-1-a1b2c3d4

# On the GKE node
$ toolbox

Step 4. Performing the Capture

Run tcpdump inside the toolbox shell. The host’s root filesystem is mounted at /media/root, so save the capture file to /media/root/tmp/, which corresponds to the node’s /tmp. Use -i any to specify capturing from all interfaces and use the Pod’s IP address, confirmed in Step 2, as a filter.

# Inside the toolbox shell
$ tcpdump -i any -w /media/root/tmp/node_capture.pcap host 10.1.2.3

Step 5. Retrieving the File

Exit the toolbox shell (exit) and then the SSH session (exit), and copy the file to your local machine using gcloud compute scp.

gcloud compute scp --project <your-project> gke-cluster-1-node-pool-1-a1b2c3d4:/tmp/node_capture.pcap ./node_capture.pcap

We haven’t had a chance to use this node-level capture in a real investigation yet, but by having the procedure established like this, we can begin investigating calmly when a problem does occur.

Summary

In this article, I introduced the practices for Kubernetes packet capturing at Mercari. Particularly at the Pod level, by leveraging Ephemeral Containers, we have established a procedure that allows developers to troubleshoot on their own while balancing security and convenience.

	Pod-Level (Ephemeral Containers)	Node-Level (Toolbox)
Primary Use Case	Investigating application-specific issues, inspecting mTLS traffic	Investigating node-wide network issues (e.g., CNI, iptables rules)
Required Permissions	Relatively low (Pod-level permissions)	High (Node SSH access)
Traffic Visibility in Istio Environments	Can capture unencrypted, plain-text traffic	Can only capture encrypted traffic
Ease of Targeting	Easy to target traffic by attaching directly to the Pod	Relatively difficult to isolate traffic for a single Pod among many
Recommended User	Application Developers, SREs	Platform Teams, SREs
Self-Service Suitability	High (Developers can investigate on their own)	Low (Limited due to the need for high privileges)

I am also pleased to announce that I will be presenting a deeper dive into this subject at SRECon26 Americas next March. My session is titled "It’s Not Always the Network (But Here’s How to Prove It): Kubernetes Packet Capture for SREs," and I hope to see some of you there in Seattle.

The next step after capturing packets is the phase of actually analyzing the captured data. Due to space constraints, and also because I’m still learning in that area, I didn’t touch on it this time, but I hope to share some knowledge on that in the future.

Thank you for reading to the end. Tomorrow’s article will be "Accelerating AI-Native Development with the Introduction of AWS Kiro and Automating Account Management with Okta" by amenbo-san and siroken3-san! Please continue to enjoy the series.

Building a Learning Culture with DevDojo

Wed, 17 Dec 2025 10:30:52 GMT

Hi! My name is Mariz, a project manager at Mercari Engineering Office. In my role, I design business processes and services that support engineers in their growth. One of the programs that I manage is DevDojo, Mercari’s unique training program for engineering new graduates and beyond into our engineering teams.

In this blog post, I’ll take you behind the scenes, and share how DevDojo has become one of the ways we build a meaningful learning culture at Mercari.

What is DevDojo, and what makes it special?

“DevDojo” combines the words Development and Dojo, with Dojo meaning “a place for immersive learning” in Japanese. When new graduates join Mercari, they can look forward to entering DevDojo – a unique training program for engineering new graduates, built by the engineers who work on Mercari’s products every day.

DevDojo has garnered a lot of interest from core engineering teams, so it has also since been made available to existing members. It has evolved into a collaborative ecosystem rather than a traditional training program, that allows engineers to personally design courses, hands-on activities, and guide new graduates through the foundational skills needed to thrive at Mercari.

We have a list of learning materials open to the public here.

From a simple idea to a structured curriculum

When my team at the Engineering Office began designing these courses, we recognized that while AI is becoming more useful in our daily work, it cannot replace the emotional dimensions of learning. Attention, memory and motivation aren’t just cognitive functions, they are shaped by human connection.

This is why as project managers of the training program, we didn’t simply create the courses ourselves or outsource them. Instead, we asked engineers to teach the next generation.
What started out as a small collection of sessions quickly grew into a structured curriculum.

We collaborated with engineers across different tech domains to design these courses based on real learning needs: fundamental knowledge for new graduates, things they wish they had learned earlier, and common issues encountered in daily development. Each course is unique, and reflects the actual practices and challenges that engineers face in their teams.

How engineers build and deliver courses

DevDojo may look like a straightforward series of courses, but the work behind each one is extensive. Engineers begin by identifying the essential concepts new graduates should understand. They draft explanations, design hands-on tasks using real system components, and we help them to map out a learning path.

Much of this preparation happens through recurring discussions with fellow engineers. It can be challenging to determine the appropriate level for new graduates and make sure they don’t feel overwhelmed.

These conversations often lead to improvement in course materials and engineering practices, as our instructors personally guide each session, and use their own professional experience to illustrate how engineers work at Mercari. Their personal insights add depth and character to every course.

DevDojo is not a static curriculum!

After every onboarding cycle, we gather feedback from both the instructors and participants. Instructors are able to run retrospectives and further refine their course materials. We send out surveys, and consolidate the data into a Looker dashboard where insights are made accessible to everybody for ongoing improvement.

This feedback loop is essential. It strengthens the identity of DevDojo as an evolving system, rather than a fixed curriculum. With every iteration, courses adapt to actual usage and remain relevant. Apart from that, we add or remove courses based on real world needs and organizational direction, ensuring the program stays current.

Building Culture through DevDojo

The most striking outcome of DevDojo is not the program itself, it is the culture that has formed around it. When engineers take ownership of designing and teaching courses, they demonstrate to new graduates that learning is a shared responsibility. Teaching becomes a part of engineering identity.

For new engineers joining the company, this sends a powerful message; at Mercari, knowledge is open and shared. For instructors, DevDojo becomes one of the ways to shape the next generation of teammates. This shared ownership is what gives our training program that culture-building power. As DevDojo evolves, one of the most inspiring outcomes we’ve observed is how it brings engineers together across teams and domains. The onboarding period becomes more than just training, it becomes a space where new graduates begin to cultivate understanding on how Mercari thinks, communicates, and solves problems collectively.

Before sessions officially begin, instructors gather for a kick-off session where a panel of experienced instructors shares practical insights from past DevDojo cycles. Topics include what has worked well, common challenges, and effective ways to engage with new graduates. It’s a space created for alignment and sharing ideas, helping all instructors feel more prepared and confident as they design courses and guide learners through the program.

Instructors often share real stories during sessions as well. Incidents, migrations, architectural debates, or moments of unexpected teamwork. Apart from being technical anecdotes, they carry the decision-making principles and values that shape Mercari’s engineering culture. New graduates don’t just learn our systems, they learn how Mercari engineers approach challenges together.

Where we go from here

As DevDojo continues to mature, the question we continually ask is not “What new courses should we add?” but rather, “How do we continue to provide a meaningful program to all engineers?”

Looking ahead, we want to preserve this cycle of shared ownership. That means continuing to refine the program based on real engineering needs, creating spaces where instructors can collaborate across domains, and ensuring every new engineer feels connected not just to their team, but to the broader Mercari ecosystem. We also want to keep DevDojo’s bottom-up culture alive, and support instructors in experimenting with new ways of teaching. Apart from that, we’ve begun using AI to collect suggestions from across the organization, helping us identify themes and learning needs more quickly.

Ultimately, both the culture and trajectory of DevDojo are rooted in the people. The program works not because it’s already perfectly structured, but because engineers choose to shape it together. We want DevDojo to remain a place where shared language is built, new engineers meet future collaborators, and teaching becomes a natural extension in their daily work. The more engineers are able to contribute, the stronger our community becomes, and the more knowledge flows across boundaries rather than staying siloed.

Designing “Mandates” for Safe and Flexible Recurring Payments

Wed, 17 Dec 2025 10:00:15 GMT

Hello, I’m @tomo, a software engineer working in the Payment Core team at Merpay.
This article is the entry for Day 17 of Merpay & Mercoin Advent Calendar 2025.

The Shift from One-Off Payments to Continuous Payments

Until now, Merpay’s payments have mainly been one‑off payments: you shop on Mercari, or you take out your smartphone, show a barcode, tap a button, and the payment is completed. However, as the Mercari ecosystem has expanded, the nature of payments has been changing significantly.

Take Mercari Mobile, which we recently launched, as an example, customers don’t open the app every month just to pay their usage fees. The system executes payments autonomously in the background.

This shift means that payments are no longer isolated events. Instead, they increasingly take the form of off-session payments—recurring charges that are executed without customer interaction, similar to subscription billing.

To support these ongoing payments at scale—and to integrate the diverse payment methods unique to Mercari—we developed a new foundational concept called the Mandate.

What Is a Mandate?

The word “Mandate” might not sound very familiar in everyday life, but in the fintech domain it’s a common term that refers to things like direct debit instructions or consent for automatic withdrawals. Similar mechanisms are provided by payment platforms worldwide, such as Stripe’s SetupIntent or India UPI’s AutoPay.

A familiar example would be a video streaming subscription service.
When customers sign up, they register their credit card information and grant broad permission like “You may charge this card a fixed amount every month.” This comprehensive consent for future payments is precisely the essence of a Mandate. These kinds of payments where charges happen later, without the customer actively interacting at the moment of charging are generally known as off-session payments.

Mandates in the Mercari follow the same idea. They represent a digital contract in which a customer authorizes a partner (for example, the Mercari Mobile service) to “use my Merpay balance, points, etc. to make future payments.”

In typical implementations, a Mandate is tied one‑to‑one to a specific credit card or bank account. For instance, you might create a Mandate to pay for subscription A using credit card B.

However, Mercari customers have a variety of payment methods, such as:

Sales / Funds
Free points / Prepaid points
Deferred payment
Credit cards

When customers pay on Mercari, they often want to combine several payment methods. For
example: “If I have enough points, use those first. If not, use my balance. And if that’s still not enough, use the credit card.” To realize this kind of composite payment without requiring any user action each time, simply linking a single card is not sufficient.

That’s why, in the Payment Platform, we designed Mandates so that they can be created against multiple payment methods. In other words, a Mandate is designed as an infrastructure component to safely implement Mercari‑specific requirements like “continuously charge using a combination of diverse payment methods.”

Mandates in Merpay

The Three Basic Elements of a Mandate

For off‑session payments, you can only make a correct authorization decision when all three of the following are clear: who is paying → to whom → and how they’re paying. These three elements define the scope of a Mandate.

Customer (who pays): Payer
Partner (who receives the payment): The service that collects the fee (e.g., Mercari Mobile)
Payment Method (how they pay): Any combination of points, balance, deferred payment, credit card, etc.

By expressing a Mandate as a combination of these three points, we can avoid granting unnecessary permissions, ensure explainability that stands up to audits, and still make payment authorization decisions in a fully mechanical way.

Mandates are managed by the Wallet Service. The Wallet Service is a foundational component responsible for managing customer-specific settings and payment permissions, such as Anshin Payment Settings.

Required Mandate Verification by the Payment Service

Off‑session payments do not involve customer interaction, so safety is paramount. We must absolutely avoid situations where a payment is mistakenly executed without a Mandate or outside the Mandate’s scope.

To guarantee this safety, we integrated Mandate validation logic directly into the payment creation API (CreateCharge).
The client calls CreateCharge with mode=off_session to indicate that the charge is being executed off-session. There is no need to check for the existence of a Mandate beforehand.

When the Payment Service receives mode=off_session, it synchronously calls the CheckMandateExistence API of the Wallet Service to validate that a Mandate exists. If a Mandate exists and is valid within scope, the payment is executed; otherwise, the process is immediately aborted and an error is returned.

By having the platform function as a gatekeeper in this way, we achieve robust safety that does not depend on how each individual service is implemented.

Delivering a Mandate‑Free Developer Experience with the Checkout Solution

With CreateCharge in off‑session mode, clients can use the API without being aware of Mandates. However, during service sign‑up, they still need to implement calls to Mandate‑related APIs. In other words, service‑side engineers must understand and implement the specification for the entire Mandate lifecycle.

To address this, the Payment Platform set out to provide a Mandate‑free developer experience by integrating with our Payment Checkout Solution so that clients no longer need to care about the Mandate API specs at all.

Merpay’s Checkout Solution was originally developed as a mechanism that provides a common checkout screen for payments. Product teams no longer need to implement payment UIs or 3DS flows individually; the platform side offers them in a unified way.

This time, we introduced a new setup mode into the Checkout Solution so that it can centrally manage the registration flow for payment methods as well. When a customer registers a payment method for a service via setup mode, the Checkout Solution internally calls Mandate‑related APIs and creates or updates the Mandate in the Wallet Service.

As a result:

To let customers configure a payment method, clients just call Checkout Solution.
For recurring billing thereafter, they only need to call CreateCharge with mode=off_session

Mandate validation and scope checks are enforced on the Payment Platform side, so clients are completely freed from dealing with detailed permission-management logic required at charge time.

How This Works in Practice for Mercari Mobile

The integration of Mandates and the Checkout Solution is already in production for Mercari Mobile payments using credit cards.
When signing a service contract, customers go through the Checkout screen once to register a credit card as their payment method. After registration, the credit card is internally linked to a Mandate, and monthly charges are then processed automatically in off‑session mode. The customer does not need to perform any special actions each month.
For Mercari Mobile developers, this also means they are freed from having to implement complex card registration flows or heavy Mandate management logic. Secure recurring billing can be achieved with a minimal implementation:

Use the Checkout Solution at contract time
Call CreateCharge in off_session mode every month

Conclusion

In this article, I introduced Mandates as the foundational mechanism for managing future payment authorizations against the backdrop of Mercari’s diverse payment methods and complex business requirements. I also showed how we integrated Mandates into the Checkout Solution (setup mode) in a way that makes them essentially invisible to both customers and developers.

Tomorrow’s article will be written by @Minato. Please look forward to it!

Supercharging User Engagement: How Mercari is Using Server-Driven UI to Reduce Time-to-Market

Tue, 16 Dec 2025 10:00:08 GMT

This post is for Day 16 of Merpay & Mercoin Advent Calendar 2025 , brought to you by @Stefan_droid from the Merpay Growth Platform team.

Introduction

The Growth Platform team in Merpay is responsible for marketing and incentive related development across the entire company. Over the years, we have built an internal customer relationship management (CRM) system, called Engagement Platform (EGP), that allows us to publish campaigns, coupons, and notifications effortlessly to our users and engage with them effectively.
This article explores how we used server-driven user interface (SDUI) architecture to implement a distributable content type called EGP Card within EGP—allowing us to supercharge user engagement while significantly reducing development effort and improving time-to-market.
EGP Card offers a flexible solution that accelerates user engagement across various campaigns and use cases by enabling remote configuration while preserving native performance and aesthetics.
This post will outline the development process, explain how this architectural shift was crucial for enhancing user engagement, and discuss common challenges encountered with such a feature, along with the solutions we devised.

The Traditional Approach & Pain Points

Super apps like the Mercari App offer many opportunities to engage with users and motivate them to take specific actions using incentives communicated through campaigns, usually displayed in various positions within the app.
Back in 2019, we introduced a 3rd-party CRM system into our app. The framework offered only very simple UI components out of the box, so we started to heavily customize our integration. Due to the various designs required for campaigns and their different locations in the app, we couldn’t rely on any default UI components, and we also didn’t have a design system back then. We ended up simply configuring key/value pairs within campaigns on the CRM and delivering them to our campaign areas inside the app.

The UI and business logic of each campaign area had to be implemented separately on every platform. We tried to promote reuse of UI components for campaigns as much as possible, but marketers frequently required changes to adjust the UI to match the next campaign. Over time, the implementations became more and more complex, more and more conditions were required to be configured with the campaign, and the apps needed to deal with all kinds of configuration combination variations. This increased the risk of incidents and the time the QA team would require to confirm the functionality of a feature. Also, for each change or bug fix, another app release was required, which made the process very time-consuming. In conclusion, creating a new component and applying changes to existing components would require almost the same amount of time and effort.

With the introduction of a Design System in 2021, we believed that would help with the standardization of marketing-related UI components, promote reusability, and reduce implementation efforts. In practice, however, marketers still frequently needed solutions beyond what any standardization or design system could offer to engage users effectively, leaving our pain points unaddressed.

What is Server-Driven UI?

This section contains an introduction to Server-Driven U. If you are already familiar with the concept, you can move on to the next section.

Server-Driven UI (SDUI) is an architectural pattern where the server dictates the structure and content of the user interface, rather than the client application (like a mobile app or web front-end) having its UI hardcoded.

The key mechanics of SDUI are:

The Server sends data and layout instructions: The server responds to a client request with a JSON payload (or similar format) that describes which UI components to render, their properties (text, color, image URLs), and their arrangement.
The Client renders UI dynamically: The client application (e.g., iOS, Android, or Web) acts as a universal renderer. It reads the server’s instructions and dynamically constructs the UI using its often pre-defined set of components (often based on a Design System).
Decoupling UI configuration from client deployment: SDUI separates the UI structure and content from the client application code itself, allowing product teams to update layouts, flows, and content by changing the server response without requiring a new client application release or app store approval.

SDUI in Action: Reducing Time-to-Market

We adopted the power of SDUI and created a feature called "EGP Card," which is one of the content types configurable for campaigns that can be delivered by our internal CRM Engagement Platform (EGP) to the client applications. This allowed us to include this new approach alongside existing alternative content types, like the previously frequently used hard-coded UI components, to quickly make it available to all clients and use already existing tooling like EGP’s WYSIWYG editor to design EGP Cards in a visual editor without spending additional effort building such an editor first.

Initially, to become production-ready, the effort required was quite high to build client-side renderers for all platforms that could render the JSON schema reliably. However, as a result, we were able to optimize our release workflow for new campaigns significantly.

Compared to the traditional approach, EGP Card’s integration is slightly faster for completely new campaign areas because of the standardized implementation that can be reused across different screens. The effort of implementing each UI element and aligning with design is unnecessary during the development phase, because the UI will be created in the web editor. EGP Card consists of a single placeholder view that renders native UI during runtime, so only the implementation of this single view and business logic is required as initial setup.
Once the campaign area is implemented, changes to the UI due to new requirements can be made completely remotely and no longer require client-side implementations or waiting time for the next app production release. As a result, the most time-consuming part becomes the finalization of specifications and design. Creating a new EGP Card or applying changes to existing ones can be done easily by drag-and-drop in the web editor and takes only a few minutes.
With this approach, it became extremely easy for us to conduct A/B tests with several variants and test which UI works best to engage with our users. New releases and updates can be published with a single button click instantly. This allows us to be flexible and react to issues immediately.

Implementation Challenges & Solutions

While Server-Driven UI offers substantial benefits in reducing time-to-market, the path to a robust and scalable SDUI system is not without its hurdles. Our experience highlighted several key challenges and led us to the solutions described below.

Versioning and Backwards Compatibility

A major risk with SDUI is introducing a new component or changing a schema in a way that breaks rendering on older client application versions that are still in use.

Solution: Client-side renderer and the overall SDUI schema are rigorously versioned.

Graceful Degradation: Client renderers are aware of their latest supported schema version and are built with robust error handling to skip rendering of schemas with higher versions to avoid application crashes. Even when unsupported components are accidentally served to an old app, the renderers will catch them during the parsing process, so that the core functionality remains stable.
Server-Side Logic: The server identifies the client’s renderer version in the request and only serves content that the specific client renderer version is known to support. Our editor allows us to specify the schema version to optionally provide different schemas to specific renderer versions, such as those used in older versions of the application.

Rendering Differences Between Platforms

The decision to go with native code client-side rendering engines for our SDUI solution came with a high cost of building specific rendering engines for all platforms that we support (Android, iOS, Web, Flutter). This was a great challenge for the team to ensure that each platform would respect and interpret all styling properties and components in the same way to ensure consistent rendering on each device.
Spoiler: We did encounter several problems with inconsistent UI rendering.

Solution: Gradual improvements, detailed documentation, and thorough testing.

Fixes & Documentation: No product is perfect from the beginning, so we continued to improve our solution and documentation to specify behavior more precisely over time.
Unit & Screenshot Testing: Before even the first production release, we created a base set of unit and screenshot tests for the base components and styling. Modern frameworks like Jetpack Compose and SwiftUI make it very easy to build and test UI, and I think that’s why SDUI has become more popular again in recent years. Thorough testing was very important as it gave us and other stakeholders confidence in our solution. We started to share JSON test cases between the platforms to ensure that our logic and behavior were aligned.
Automation: Several changes and improvements over time can easily cause regressions. To avoid that, we integrated our screenshot testing into our CI/CD workflow. Furthermore, our team built tooling that allowed us to compare all platforms directly with each other to quickly discover differences (see screenshots below).
Utilize AI: AI is a great help for finding issues, improving rendering logic, and creating comprehensive test cases. For Mercari Hallo, an app built with Flutter, we even created a native Dart plugin to support EGP Card instead of building a plugin using native Android and iOS channels in the background. The reason for that was a mix of dependency issues, complexity, and a tight deadline by which we needed it. Luckily, building an additional renderer becomes, thanks to AI, a very easy task. Agents can quickly understand the logic used and generate code from one language into another, and then use existing test cases to validate the logic and the rendered UI output.

Design System integration

One of the most frequently asked questions to our team was whether our solution uses our internal Design System, and people were surprised when we answered that we didn’t, at least not directly. The main reason is that the requirements we receive from marketers often don’t align with the Design System, and building a solution purely on the Design System would result in us adding more and more exceptions to satisfy the requirements.

Solution: We decided to make the SDUI styling as granular as possible and make it easy to use with our existing What-You-See-Is-What-You-Get (WYSIWYG) web editor. This option gives us the most freedom but also adds more complexity to the rendering engine. However, since we are always striving for improvements and making processes easier, we are currently planning to integrate the Design System components into our web editor by automating the generation of component-level templates using an AI agentic workflow to publish them based on our Design System.

Personalization

Marketing often wants to engage with our users as personally as possible and create personalized experiences that are most valuable to them. As a basic example, instead of just showing a generic campaign about a clothes-related coupon: "Save 20% on clothes products," we want to personalize the experience and show a liked item of the user from the clothes category and display how the price would change for them if they used the coupon. This method is much more effective and more meaningful for the user.

Solution: Introducing placeholders and compose final schema on backend

We decided on a simple {{ key }} schema that can be used across the web editor to replace predefined keys on our backend side. During the API request from the client, the backend fetches the static schema for the EGP Card from the CDN and then replaces all the placeholders with aggregated data for the specific user. This approach simplifies the client-side renderers, keeping them "dumb" and eliminating the need for complex replacement logic. The backend and web editor frontend require some kind of contract to understand which placeholders are available. Currently, we are relying on documentation to achieve this, but this could be further improved by using for example Protocol Buffers (Short: Protobufs – Google’s language-neutral data serialization format) to have a single source of truth and add new placeholders automatically.

State Management and Interactive Components

One frequent feature request is the possibility of adding interactive components to EGP Cards, such as a Like button. But even a simple Like button can become really complex when trying to design a feature that is scalable for more than a single use case. Let’s take a closer look at it:

Example requirements for a Like button:

Show different UI based on the like state (liked, not liked)
When the user taps the Like button, the state should change and trigger an asynchronous API request to the backend to persist this information
When the API request fails, the state should return to ‘not liked’

Suddenly, our SDUI features need to be able to maintain a state and make API requests. These two requirements are not trivial to address in our so far static JSON schema. It might be arguable whether such a feature belongs in a SDUI solution or whether stateful logic should rather be implemented natively.

There is probably no single best solution to this problem, but there are several approaches to deal with it.

Possible Solutions:

Custom-Components: Probably the most common solution with the least impact on the existing schema. Simply create a new component that can be selected which contains a reference ID. The client will replace the component based on the reference ID with a hard-coded component defined on the client side, which already holds all the business logic and state management that are required. Each client needs to implement the component individually; otherwise, clients wouldn’t be able to display it. Using a custom component makes it difficult for the web editor to preview its appearance unless a specific preview for each component is provided.

{
    "type": "Custom",
    "referenceId": "IconLikeButton"
}

// Android Compose Component Example
EgpCardView(
    egpCard = card,
    isDarkMode = state.isDarkMode,
    onDisplay = { ... },
    onClick = { ... },
    onNavigate = { ... },
    customComponent = {
        when (it) {
            "IconLikeButton" -> IconLikeButton()
        }
    }
)

Stateful Wrapper-Component: Another solution would be to create a new stateful wrapper component that is able to maintain a state and share it with its child components down the component tree. It would also have knowledge about the API endpoint and everything that is required for the client to compose a valid API request. Based on a successful or failed response, this stateful wrapper component could adjust its state and control the UI. This approach requires adding a very complex new component to the schema and might not work with non-REST-based endpoints.

{
    "type": "StatefulWrapper",
    "stateRef": "state1",
    "states": {
        "init": {
            "id": "clickable_element",
            "type": "IconButton",
            "actions": {
                "onClick": [
                    {
                        "type": "API/REQUEST"
                    }
                ]
            },
            "styles": { ... },
            "children": [ ... ]
        },
        "loading": null,
        "error": { component },
        "success": { component }
    },
    "api": {
        "url": "/api/endpoint",
        "method": "POST",
        "data": {},
        "headers": {},
        "onSuccess": {
            "type": "setState",
            "stateRef": "state1",
            "value": "success"
        },
        "onError": {
            "type": "setState",
            "stateRef": "state1",
            "value": "error"
        }
    }
}

Both solutions have their advantages and disadvantages. We are still considering which approach fits best for us, and there might even be a better solution.

Generate UI from Figma design using AI

Despite using a web editor for UI creation, which enables continuous deployment, the development of the UI components themselves still takes place within the editor. To significantly shorten the cycle from design concept to deployable components, we investigated the use of modern AI models to assist in this process.

Solution: Initially, we considered utilizing AI models and MCP to generate our schema based on design tokens from Figma. However, we require consistent output based on the same input data, and most modern AI models do not offer the ability to control their temperature (setting to control randomness in LLMs). Therefore, we decided to develop a plugin for Figma instead to get deterministic results. Designers could define a new component in Figma, and the plugin would automatically generate a preliminary schema based on the design tokens and structure. This schema could then be imported directly into the web editor. While it doesn’t fully automate the process, it significantly reduces the manual effort required for the initial implementation of a new component.

Conclusion

SDUI solutions require a lot of initial effort and some risk-taking to invest the time to achieve a production-ready solution. Our Growth Platform engineering teams believed in this solution, which helped us gain trust and resources to build it. Now we frequently get inquiries from other teams asking us about the feasibility of building their features with our solution. This helps us improve and extend it further and MOVE FAST together. Modern frameworks, languages, and increasing internet speed significantly contribute to the success of SDUI solutions, and we can see that this will become increasingly relevant technology in the future, especially for marketing, where speed and flexibility are key.

The Cost of Speed: A Battle against Cost, Debt, and Diverging Systems

Tue, 16 Dec 2025 09:00:49 GMT

This post is for Day 16 of the Mercari Advent Calendar 2025.

Introduction

Hello, my name is Sneha. I am a Director in Product Engineering, managing the Ads and Shops product engineering teams. I want to share a personal journey—not just of systems and code, but of a “perfect squad” known as the Shops Enabling Team.

What follows is a journey of resilience.

Three engineers.
Two incompatible systems.
One year to fix spiraling costs.

It is the story of the Enabling Team—with a massive challenge: merging two heterogeneous systems to reduce operating costs and stabilize the Mercari Shops systems. The work continues, but the most challenging part is behind us.

The Origin: Going Bold and Drifting Apart

It all started with a notification that is very common within engineering organizations in any company: “We don’t have enough people to maintain the current systems, and our systems are becoming ‘too expensive’ to run”.

To understand why, we have to look back a bit. When Mercari decided to “Go Bold” and invest in growing our B2C business, we launched Mercari Shops (aka Souzoh Inc.). The directive was clear: Validate the new business hypothesis fast.

To unlock this velocity, we made the strategic choice to break away from our core foundation and platform services. We chose a stack optimized for speed and also improved the system design by learning where our existing architecture failed to deliver. We bet on Cloud Run (Serverless) to keep ops overhead near zero and used Bazel to tame our monorepo of 80+ microservices. With gRPC for backend traffic and Next.js on the frontend, we built a system optimized purely for speed, allowing us to focus on product features rather than platform maintenance.

It worked. We shipped fast, operated as a single small unit, and the business numbers climbed.

Then the product direction shifted, marking a new chapter in our journey!
We wanted to provide a seamless, unified experience, effectively erasing the boundaries between Business (B) and Consumer (C) sellers for the users.

This shift confirmed my core philosophy: “a system is a living ecosystem”. If the business evolves, the architecture must evolve with it.

Engineering found itself in the middle of a massive reconciliation problem. We were maintaining two heterogeneous systems that were similar in many respects. We built “bridges”—glue code—to force them to work together. As the years went by, the system grew bulky. Latency spiked, customer UX deteriorated, and complexity soared. And finally, the Cost Per Transaction (CPT) hit a breaking point.

The Breaking Point: The “Shops Enabling Team”

We needed to reduce costs and complexity, but we were stuck. Internal discussions revealed that a standard “fix” would require a major refactor, which would stop feature development work for almost 2 years. For a growth business, that was impossible!

For over a year, we debated. The debate centered on a crucial question: “Should we align Mercari Shops systems with our core services environment?” While the answer seemed to be ‘yes,’ the execution required such immense effort that we struggled to commit to a unified vision. We needed a strategy that would unblock business growth while handling years of accumulated debt.

The breakthrough came in July 2024. We moved from meetings, offsites and discussions to focused execution by establishing the ”Shops Enabling Team”.

The team’s goal was simple yet critical: “dismantle the obstacles holding us back, one by one”.

It was small—just three engineers—yet they formed the essential bridge spanning across various engineering organizations within Mercari.

The Operational Blueprint: Strategy, Synergy, and Speed

The formation of this team marked a cultural shift. To succeed, we had to change how we operated fundamentally:

Strategic Architecture: The Principal Architect in the team devised a strategy rooted in reality, not theory. We accepted that the ‘perfect world’ solution is a myth; real progress happens in iterations. It helped us avoid numerous discussions that weren’t addressing the problem.
Embedded Synergy: We embedded engineers from different platform domains into the team, cutting through the organizational ’telephone game’ to align priorities instantly.
Strategic Rituals: The shift in the standups. Standup’s were no longer about the usual 1-line status updates (“I did X yesterday”). Instead, they became strategic war rooms where the team discussed how to solve the day’s blockers and architectural hurdles. As an EM, these became the most insightful and productive meetings of my day. I learned a lot!!!
The Feedback Loop: The Shops Enabling Team was right in the middle of Platform Engineering and Product Engineering orgs. We created a continuous feedback loop to identify the strengths and weaknesses of each side. It didn’t just help Mercari Shops systems; the feedback we collected fueled improvements back into the core Platform, benefiting the entire company.

The Audit: The Low-Hanging Fruit

We started looking at our GCP bills. A deep dive into our GCP components revealed the usual suspects: duplicate data pipelines running in parallel, unoptimized services burning unnecessary CPU cycles, and so on.

We fixed these quickly, feeling a momentary sense of victory as the Cost Per Transaction (CPT) dropped by 20%. But the celebration was short-lived.

The data made one thing clear: we had exhausted the easy fixes. To reach our goals, we would have to stop avoiding the ‘tricky and hard bits’—the messy, complicated architectural debt that we had been too afraid to refactor.

The First Tricky Bit: Convergence through Unification

Our users don’t distinguish between ‘B’ and ‘C’ items on their screen, so why should our backend?
Recognizing that the C systems already offered a mature, feature-rich Search & Recommendation engine, we initiated a strategic merger of our Search and Recommendation systems rather than reinventing the wheel.

We decommissioned the entire Mercari Shops-specific search and recommendation infrastructure, including shutting down costly Vertex AI and Elastic Cloud instances.
We adapted the common search components that supported logic for B items within a unified “B & C Search and Recommendation” framework.
This consolidation enabled new search features to launch simultaneously across both Mercari and Mercari Shops.
It was a win-win situation in product and engineering!!

It wasn’t an easy win. We underestimated the depth of the cleanup required, discovering layers of technical debt that needed to be tackled before we could move forward.

[Note: For a deep dive into the code-level challenges and how we solved them, check out this article by one of the Enabling team engineer.]

The architectural cleanup delivered immediate cost efficiencies, slashing the Cost Per Transaction (CPT) by 12.5% for cumulative savings of 30%.

Our dramatic drop in system costs didn’t go unnoticed. It triggered a spotlight moment: our Internal FinOps team reached out, not to audit us, but to collaborate. With their recommendations unlocked , a further 9.5% improvement, culminating in a total Cost Per Transaction (CPT) reduction of 36.7%.

But the true victory wasn’t just the number—

It was the decoupling of cost from growth. Even as Shops’ business surged (increased in the number of transactions), our costs remained flat. The ‘fixes’ held firm, proving we had finally broken the cycle of linear cost scaling.

The Second Tricky bit: Moving to GKE without breaking DX

Then we moved our attention to the infrastructure layer, i.e., the Serverless architecture we chose for Mercari Shops. We were not able to scale it effectively as the business grew. We needed to move away from Cloud Run onto our unified GKE cluster. It was a dilemma on how to scale the systems for exponential growth without hitting the brakes on feature development.

This migration required us to Protect Developer Experience (DX) while doing the changes in the infrastructure layer. The team needed to really dig deeper to understand the current developer experience, which required them to interview members of the feature teams. Align on what is essential for the feature teams and what we can change.

We kept the monorepo and toolchain (Go/TypeScript/React) intact. We only shifted the operational “under the hood” components—specifically, moving logging from GCL to Datadog and deployment to WarpSpeed CD (internal tool for CI/CD ) . It minimized disruption for engineers accustomed to the existing workflow.

Instead of separate Kubernetes kits for each service, we built a single starter-kit (config) for all Mercari Shops services. It provided us with custom networking controls to build a bridge between the old Cloud Run and new GKE environments. To prepare for worst-case scenarios, we needed to build a Flexible Traffic Flow, so our principal architect designed the architecture that allowed requests to flow back and forth between the Cloud Run and GKE environments. It prevented “Big Bang” cutovers and provided an immediate, safe rollback mechanism should any unforeseen issues arise.

Behind the Scenes: Managing the Migration Chaos

The Reality of Migration – It wasn’t a smooth ride. We faced incidents, rollbacks, hidden traps, and dependencies. As we went deeper in the various service layers, we discovered inefficiencies and anti-patterns buried deep in the legacy code. We also had to familiarize feature engineers with the current infra layer. But the challenge wasn’t just technical; we were fixing context as much as code. Maintaining feature velocity required proactive knowledge transfer. We organized targeted enablement sessions to address specific blockers that feature engineering teams face.

The Shops Enabling team was effectively triaging the flood of notifications from feature teams. This hands-on support model allowed us to onboard teams quickly and resolve DX issues before they slowed feature delivery.

The “AI” Factor – Since the Shops Enabling team was new to the Mercari Shops system and did not understand the features and the services, we adopted AI tools like Cursor early on to fill the knowledge gap. We used it to analyze old documentation, Slack threads, and legacy code to get the historical context.
During development it boosted the generation of migration scripts that would have taken a week to write manually. AI became our force multiplier.

The “Perfect” Dashboard – You cannot fix what you cannot see. We realized early on that our existing monitoring was insufficient for the complexity of this migration. We took time to build the ‘Perfect Dashboard’ in Datadog—a single pane of glass that revealed the system’s heartbeat❤️. But metrics weren’t enough; we needed context. We implemented end-to-end distributed tracing, enabling us to trace every request across the heterogeneous stack and ensure nothing was lost in the transition.

However, what kept the team going, despite these challenges, was seeing the traffic graph slowly shift until it hit 100% on GKE.

In July 2025, we crossed the finish line: 100% traffic migration to GKE. YAY!!

The system stability improved, and Cost Per Transaction (CPT) dropped by 33.3% (cumulative gain 53%). We addressed an inefficiency we observed in logging and applied a quick fix, driving costs down further, achieving a massive 67% total reduction in Cost Per Transaction.

As Mercari Shops was Mercari’s first large-scale Monorepo, we encountered multiple edge cases that no other engineering team had faced. These challenges generated a lot of insights. We funneled these learnings directly back to the Platform teams, catalyzing major upgrades to our CI/CD infrastructure and developer tooling.

The Final Tricky bit: Breaking down inter-service walls

With the infrastructure settled, we wanted to resolve the last tricky bit: the Identity platform for Mercari Shops
.
The Mercari Shop’s custom token was a wall, not a bridge. Shops relied on a custom token that required maintaining years of accumulated ‘cold’ code—forgotten custom logic. It isolated Mercari Shop services and made communication with the core services outside the Shop system painful and inefficient. As our product aimed for a unified UX, this distinction made it challenging to communicate with other services, leading to messy ‘Glue code’.

We decided to stop maintaining a parallel identity stack. By adopting the Mercari PAT (Private Access Token), we not only simplified our architecture but also unlocked true interoperability with the broader Mercari backend ecosystem.

We couldn’t fix identity in a single go, so we broke the migration into two phases: Internal and External usage. We prioritized the internal cleanup first.

Upon migration to Mercari PAT, we identified two critical blockers. First, the Mercari PAT didn’t support the Google Identity Platform used by our B-Sellers. Second, Shops tokens carried custom claims that the Mercari PAT didn’t support.
We engineered a bridge in our internal auth service to convert Shops Tokens to PATs, preserving the external user experience. Simultaneously, we re-architected the dataflows to fetch custom claim data via gRPC rather than relying on the token.
It wasn’t a quick fix; it required modifying 80+ microservices.

While AI accelerated the code generation, the real battle was rigorous testing to ensure zero regressions. After a long journey of testing every use case, we decided to release.
The moment we enabled direct service-to-service calls, the benefits were undeniable.

We didn’t just simplify Mercari Shops’ system architecture; we unlocked true interoperability.

We received many requests from various engineering teams to switch to direct calls, simplifying integration across heterogeneous systems.

We are still in the second phase of the migration work. I hope we can wrap it up soon.

The Leader’s Playbook: Leading Through Legacy

As engineering leaders, we often agonize over how to rewrite our systems—which architecture to pick, which stack to use. But the truth is, the biggest challenge is rarely the system itself; it is the inertia of operating within a large organization.
Based on our journey of the various migrations that we did, here are some recommendations for leaders looking to move the needle in complex, brownfield (legacy) environments:

Design the Organization, Not Just the Architecture
In large organizations, systems often inevitably reflect communication structures, and organizational fragmentation becomes the main bottleneck for modernization. Silos prevent the cross-functional collaboration required to fix systemic debt.
The Strategy – Don’t rely on existing teams to do new tricks. We explicitly formed the "Shops Enabling Team"—a small, dedicated squad sitting across different engineering verticals.
The Takeaway – If your architecture is stuck, look at your org chart. You may need to spin up a temporary, specialized unit whose only KPI is to break silos and unblock flow.
Cultivate an “Evolutionary” Mindset
The “perfect squad” isn’t necessarily made up of the deepest experts in the legacy or latest tech stack. It is made up of engineers who are open to learning and evolving as they go.
The Strategy – The Shops Enabling team succeeded not because they knew everything from day one, but because they were resilient enough to learn many things on the fly.
The Takeaway – When staffing a modernization team, prioritize adaptability over tenure. You need people who view the system as a living ecosystem, not a static monument.-
AI is the Bridge from Brownfield to Greenfield
We are entering a new era of software development where the economics of refactoring have changed. The cost of transforming ‘brownfield’ legacy systems into ‘greenfield’ modern architectures is reduced and it is no longer manual work—it is an AI-assisted acceleration.
The Strategy – We used AI tools not just to write code, but also for “software archaeology”—analyzing legacy documentation and running various simulations to assess risks.
The Takeaway – Stop treating AI as just a coding assistant. Use it as a force multiplier to de-risk the most dangerous part of migrations: the knowledge gap.

The Hidden Wins and Personal Reflection

The impact of the work scaled far beyond the three core migrations. We optimized our caching layers and resolved critical database inefficiencies, and slashed onboarding costs by standardizing infrastructure.

Overall, it was a huge win:

We halved our system costs, reducing Cost Per Transaction (CPT) by a massive 67%, even amid rapid business growth.

Yet, the real victory was the journey itself. It reignited a spark I hadn’t realized was dimming. Reconnecting with the roots of engineering — not just managing it, but feeling the daily reality of it — ultimately made me a better leader.

None of this would have been possible without the Shops Enabling Team and the cross-divisional trust the team built among other engineering teams.

With the right strategy, people, and organizational setup, you can do the impossible: rebuilding your core infrastructure in mid-air, making it cheaper, faster, and better without ever touching the ground! 🚀

Tomorrow’s article will be by mariz about Building a Learning Culture with DevDojo. Stay tuned!

Extending the Balance Service: Challenges in Implementing Multi-Currency

Mon, 15 Dec 2025 10:00:54 GMT

This post is for Day 15 of Merpay & Mercoin Advent Calendar 2025 , brought to you by @timo from the Merpay Balance team.

We are responsible for the "Balance Service," which manages the ledger and booking of user funds.

In this article, I will introduce the challenges we encountered when extending our system to support multiple currencies and how we resolved them.

Background

Our system runs on a double-entry bookkeeping architecture.

When we originally designed this architecture, we defined the concept of "Exchange Rates" to support future global expansion. However, since the immediate business requirement was domestic, we only implemented the logic for Japanese yen (JPY).

This year, to support the Global Business expansion, we proceeded to implement the full multi-currency feature set. Moving from a JPY-only implementation to a system that handles multiple currencies (USD, EUR) introduced specific engineering issues related to data modeling and precision.

Prerequisites: The "Exchange" Data Model

Before discussing the challenges, it is helpful to understand our transaction structure. We define an Exchange as a single unit containing two distinct flows (Money Out and Money In).

Here is a simplified view of our gRPC Proto definition:

message Exchange {
    // Header Level: Metadata
    string transaction_id = 1;

    // Detail Level: The Legs of the transaction
    Source source = 2; // Who is paying? (e.g., TWD)
    Target target = 3; // Who is receiving? (e.g., JPY)
}

Ideally, the core logic of our system is maintaining the balance: Source Amount * Exchange Rate ≈ Target Amount. Managing this equation and deciding where to store the rate became the central theme of our challenges.

Challenge 1: Data Modeling for Exchange Rates

An "Exchange" transaction consists of a Source (Money Out) and a Target (Money In). The first issue we faced was determining where to save the exchange rate.

We considered two storage patterns:

Option A – Exchange Level (Header): This approach places the rate in the Exchange table and the API Header.

This fit our current use cases perfectly and was easy to implement. However, it was not flexible. If we ever needed to support mixed-currency payments (e.g., TWD + USD) in the future, this structure would require a difficult migration.

Option B – Source Level (Detail): This approach places the rate in the Source table and the API Source message.

It is highly flexible and easy to extend. However, for our current needs, it felt like over-engineering. Forcing the Proto to carry a rate for every single source—when we currently only do 1-to-1 exchanges—would make the API unnecessarily complicated for our clients.

Our Decision: We decided to separate the API design from the database schema.

API Level (Proto): We accept the rate in the Exchange Level. This keeps the integration simple for our upstream clients, who simply request: "Convert TWD to JPY at rate 4.2."
Database Level: We map and store the rate at the Source Level.

This hybrid approach gives us simplicity in the API and future flexibility in the database. When the time comes to support multi-source payments, our database will be ready without migration, even though our API is currently optimized for simple use cases.

Challenge 2: Precision and Validation Logic

The second issue was validation logic. In a JPY environment, the math is always integer-based (100 = 100). In a multi-currency environment, Source * Rate results in decimals.

The problem is that we cannot assume which rounding method (Floor, Ceiling, etc.) the clients use. If we enforce a strict rule (e.g., Round-Half-Up), valid transactions might fail due to minor rounding differences.

To solve this, we first clarified the responsibility of the Balance Service. It is a System of Record, not a Pricing Engine.

If the upstream service rounds down and "loses" a fraction of the value, that is a business decision made upstream. Our responsibility is not to enforce pricing strategy, but to ensure the booking is mathematically consistent within a reasonable margin of error.

The Solution: Based on this core principle, we implemented a Flexible Validation approach. Instead of checking for an exact match, we check if the Target Amount is "mathematically reasonable" given the Source and Rate.

Calculate: Compute Source * Rate.
Determine Precision: Compare the calculated result with the requested Target Amount. The value with fewer decimal places is used as the reference.
Validate: Round the precise value both Up and Down. If either result matches the reference, the request is accepted.

This allows us to accept valid transactions regardless of the upstream rounding method (Floor, Ceiling, Banker’s Rounding) while still blocking truly incorrect rates.

Challenge 3: Designing the Reversal Interface

The third point is a design insight regarding our existing Reversal API.

We already had a stable Reversal endpoint. With the introduction of multi-currency support, we faced a question: Should we add an exchange_rate field to the Reversal request?

Ideally, a Reversal operation should only output the original exchange rate (for reference), never take it as input.

If we added an exchange_rate field to the input, it would create confusion for the client: "Should I send the current market rate or the original one?" If they accidentally sent the current rate, the ledger would become unbalanced.

Our Approach: We decided to use exchange_rate as output-only in the Reversal API. The system internally looks up the original rate to ensure the "Undo" is mathematically exact. By limiting the input schema, we prevented the possibility of rate fluctuations errors by design.

It is worth noting that, if a Business Refund is required (where the refund is based on the current market rate), this does not need a special endpoint. It can be implemented simply by calling the Exchange endpoint, swapping the original Source and Target, and providing the new rate.

Future Issues: The Attribution Problem

Looking ahead, we anticipate complexity with Multi-Source Payments. If a transaction uses multiple sources (e.g., TWD and USD) to pay a single JPY target, rounding errors may cause the sum of the converted amounts to differ from the total target amount. Determining how to assign this rounding difference (which source absorbs the gap) to maintain a balanced ledger is a topic we recognize as a future challenge that we will need to solve.

Extra: The Regional Constraint

Finally, I want to touch upon a design requirement that often comes up during global expansion.

In the initial design phase, adding a Currency field seems like the only requirement. However, a critical realization often follows: Currency and Region have a many-to-many relationship.

A single currency code does not uniquely identify the legal region. For example, USD is the official currency of the United States, but it is also used in other places (like Ecuador). Conversely, a single region like Panama uses both PAB and USD as official currencies. (List of circulating currencies)

Legally, "Region A-held USD" is different from "Region B-held USD" due to different financial rules. Since the currency code is the same, we cannot tell them apart by Currency alone.

Therefore, we established Region as a required dimension in our ledger. By managing assets based on the combination of Region + Currency, we ensure that funds remain correctly separated across different regions.

Conclusion

Supporting multiple currencies required more than just adding a currency field. We had to finalize the data model, implement flexible validation for decimals, and strictly define the scope of reversals.

By addressing these specific challenges, we were able to support global business requirements by simply extending our existing architecture, without the need for a fundamental redesign.

In this post, I shared how we extended the Balance Service and the solutions we used to handle multi-currency challenges. I hope this gives you some new ideas for your own system designs! 🙂

Tomorrow’s article will be by @Stefan_droid. Look forward to it!

Search Results Quality Monitoring with LLMs

Tue, 09 Dec 2025 11:00:59 GMT

Hello, I’m @otter, a software engineer working in the search domain at Mercari.
This article is the entry for Day 9 of the Mercari Advent Calendar 2025.

Mercari’s Product Search and Its Quality Management

Mercari’s product search plays a crucial role in accurately understanding our customers’ intentions among a massive number of products and displaying the exact items they’re looking for in the search results. Therefore, it is essential to continuously check the relevance and validity between search keywords and search results to maintain and improve quality.
In this article, I will introduce how we have leveraged LLMs (large language models) to improve the quality check flow for search results.

Challenges and Requirements in Search Results Quality Review

Until recently, product managers and engineers had to visually check each search result item sampled for different keywords and calculate the proportion of irrelevant items. This manual process was extremely time-consuming, and also led to inconsistencies and instability in evaluation results when done by multiple people due to variations in evaluation criteria.

In light of these challenges, our quality review process now needs to be automated on a daily or weekly basis, monitored through a dashboard, ensure a consistent and sufficient volume of reviews, include clear evaluation criteria, and accurately capture the context and intent behind users’ searches.

Achieving Objective and Stable Monitoring with LLMs and Evaluation Criteria

To meet these requirements, we implemented several LLM-based quality reviewers for search results.

After comparing several LLM models, we decided to leverage Gemini 2.5 Pro as it best understood users’ intent through the experimentation phase.

At first, we evaluated search results by providing only screenshots of the results pages to the LLM, simulating the user’s perspective. However, with this approach, it was difficult for the LLM to make judgments that accounted for detailed product information, leading to misclassifications, for example, due to differences in product specifications or categories. To improve the accuracy of the evaluations, we modified the process to also provide the LLM with detailed information for each item, such as the product name, type, price, category, and thumbnail image.

Evaluation Criteria

We instructed the LLM to return a "Relevance Score (0.0–1.0)" and a rationale for each item. The scoring is based on Amazon’s ESCI relevance judgements (Exact, Substitute, Complement, Irrelevant), with scores assigned to each class:
　

Exact (1.0): Products that perfectly match the specified query (e.g., "iPhone 14 Pro Max 256GB" → the exact model and specification)
Substitute (0.75): Products that are functionally usable as substitutes (e.g., "iPhone 14" → iPhone 13; similar specification but different generation)
Complement (0.5): Accessories or complementary products (e.g., "iPhone" → iPhone case, charger)
Irrelevant (0.0): Completely unrelated or not meeting the requirements (e.g., "telescope" → socks)

With our previous manual evaluations, the assessment criteria tended to be subjective, often resulting in inconsistent outcomes. However, by introducing clear scoring definitions and leveraging LLMs, we have significantly improved the stability and objectivity of our evaluation results.

How the Quality Monitoring Tools work

For our search team, there are currently two major use cases for Search Relevancy quality checks.

Online Monitoring

We randomly extract search keywords from production search query logs and evaluate the relevancy of their results. Every week, about 1,000 keywords are sampled, and for each, the top 120 items in the search results are reviewed.
Review results are output to a BigQuery table and can be routinely checked through a monitoring dashboard, etc. When conducting A/B tests for search quality improvements or releasing new features, we can monitor changes in metrics such as Average Relevance Score or Irrelevant Items Rate.

Offline Evaluation

We also use it for offline evaluation before running A/B tests on new features or for improvement validation. By entering keywords to be examined, engineers or product managers can instantly see the search results, category/brand/price distributions, and LLM-based evaluation results via a tool. It’s also possible to conduct large-scale batch reviews using pre-determined keyword sets.

Although these two use cases run on different systems, by unifying the LLM prompts, we ensure consistency in the evaluation criteria and results.

Possibilities for Further Expansion

Combining image data with text data has improved evaluation accuracy. However, there are still challenging cases that require human judgment. That said, model accuracy continues to improve drastically every year, and we expect even further automation in the future.
Additionally, beyond evaluation and monitoring, we are also considering using LLM-generated evaluation data itself as training data to improve the underlying search models.

Conclusion

In this article, I introduced our efforts to automate, stabilize, and improve the efficiency of the evaluation of search result relevance through the use of LLM, which had until now relied on human review only at Mercari.
The introduction of LLMs has led not only to more efficient review operations at Mercari but also to the realization of continuous quality monitoring based on more objective evaluation axes.
Going forward, we plan to further improve our search features by leveraging evaluation data and addressing even more difficult cases.
I hope this article proves useful to those struggling with quality evaluation in search or recommendation systems, as well as those interested in utilizing LLMs.

Tomorrow’s article will be written by @task. Please look forward to it!

Navigating Change: Learning to Reinvent in an Unstable World

Mon, 08 Dec 2025 11:00:43 GMT

My name is Antony Chane-Hive. I joined Mercari in 2018 as an Engineer and I’m currently an Engineering Manager in the Product Engineering division.
In this article, I will explore certain themes such as Psychological Safety, Self-Determination Theory and Dynamic Reteaming.

This post is for Day 8 of the Mercari Advent Calendar 2025.

(~15 min read)

The Paradox

When my manager announced a new reorganization, I caught myself having two opposite reactions at the same time.

Part of me felt curious.
A new team meant new challenges, new problems to solve, new people to learn from. I’ve always liked change. Chaos brings opportunities. It forces growth.

Part of me felt exhausted.
The fatigue of constantly starting over. Of being a novice. Of rebuilding relationships, relearning processes. Again.

How do you love chaos and be tired of it at the same time?

That’s when I realized something I’d been avoiding: I was waiting. Waiting for the next change to be the last one. Waiting for things to finally stabilize. Waiting for the ground to stop shifting so I could finally build something that would last…

I was waiting to rebuild my comfort zone. To feel competent. To know who to ask for help, how things worked, where I fit.

But that moment wasn’t coming. Maybe it never would.

The pace keeps accelerating. AI is reshaping how we work. Strategies expired before we could even implement them. And it is not just us—the world itself is spinning faster. A pandemic. Political changes. The ground is changing around us, not just at work.

Instability is not a phase to endure. It’s the new operating system we need to learn.

So, I started thinking about what worked and what didn’t. I observed people who seemed to adapt faster—how they asked questions, how they built relationships, how they approached the unfamiliar. I paid attention to the forces behind the changes themselves. I experimented with tools and frameworks.

This is not a prescription for how you should navigate change, but an invitation to reflect on how you’re navigating it right now.

Because here’s what I’ve learned: We’re all learning this in real-time. Some of us are just asking different questions.

Part 1: The Safety to Not Know

Earlier in my career at Mercari, a task looked straightforward on paper. A coding implementation that needed deep domain knowledge. As a mid-career engineer with a couple of years of backend and frontend experience, I should have been able to handle it perfectly.

But I wasn’t sure I could.

In the team discussion, colleagues referenced concepts and patterns I’d only skimmed in documentation. My instinct was to nod along, to protect my image of being a mid-career engineer.

So I did. I nodded, took notes, and spent time piecing together what I’d pretended to understand. I eventually succeeded (and learned a lot), but it took longer than it should have.

Feeling confident is our accelerant for steady progress, reducing stress and maintaining our motivation.

I thought I was protecting my status. I was actually delaying my effectiveness.

The Turning Point

When the company changed my role to engineering manager, I faced a different kind of gap.

"Congratulations on your new role" they said. "You’re now the manager of this team."

My knowledge was lacking. I didn’t know where to begin, what I should do or how. And unlike a coding task, I couldn’t fake management by searching on the web.

In one of my first meetings with my manager, I asked the questions I’d been avoiding: "How do I conduct 1-on-1s? How can I grow my members? What should I look for?"

I didn’t understand all the ramifications of the answers. I had to figure it out myself.

But something started to shift.

The learning wasn’t faster because I got better answers. It was faster because I’d stopped spending energy on pretending.

Different paths, Same need

As I observed how others navigated similar transitions, I noticed people approached uncertainty differently.

Some colleagues asked direct questions in meetings. Others researched before responding. Some observed quietly, piecing together understanding through pattern recognition.

None of these approaches were better than the others. They were just different ways of managing the same vulnerability. What mattered wasn’t how someone expressed uncertainty—it was whether they felt safe doing it.

The need for belonging transcends culture, even if how we say "I don’t know" varies. The vulnerability feels the same, whether you’re asking directly, observing first, or probing the situation. We’re all managing the same question: Can I be incomplete here and still belong?

Change often feels threatening not just because it’s new, but because it can mean a loss of safety, status, or belonging. Recognizing these feelings—our own and others’—is the first step to building trust in any environment.

How I nailed it

I discovered there was research behind what I’d experienced.

Psychological safety—a term coined by Amy Edmondson—describes a shared belief that interpersonal risk-taking is safe. That you won’t be humiliated, punished, or marginalized for speaking up, asking questions, or admitting mistakes.

It’s not about comfort. It’s about learning effectiveness.

Edmondson’s research showed that teams with high psychological safety learn faster. Not because they’re smarter or more talented, but because they can be beginners without penalty. They ask questions early, when confusion is small and correctable. They experiment without fear of judgment.

The Psychological Safety Quadrant

Having a name for it—psychological safety—helped. Though, sometimes, I still go back to my flaws, pretending I understood. It’s still happening. Knowing the concept doesn’t make the fear go away. It just makes you feel more foolish when you do it anyway.

When change throws you into a new domain—whether it’s coding or management or any unfamiliar territory—everyone becomes a beginner at something. The difference between those who adapt quickly and those who struggle isn’t intelligence. It’s whether they can admit what they don’t know.

What changed

The shift from pretending to admitting didn’t make learning easier. It made it possible.

I stopped rehearsing confidence I didn’t have. I started asking questions when I was confused, not after I’d tried to figure it out alone for too long.

I watched colleagues who seemed to be adapting well and copied their approaches. Sometimes, I was a bit envious. I learned from my peers, junior and senior alike.

Each new role—engineer, manager, manager of managers—brought its own beginner moments. Vulnerability and learning are not just for newcomers; they’re part of every stage.

More importantly, I noticed something unexpected: people responded differently.

They offered information more freely. They were patient and showed empathy. Some realized they weren’t alone and began asking more questions.

Admitting ignorance didn’t cost me status. It earned me credibility.

The Foundation for everything after

Those transitions taught me: You cannot navigate change quickly without psychological safety—either the kind your environment provides, or the kind you build for yourself.

If your team creates it, use it. That trust is there for a reason. Ask those questions that feel too basic. Admit confusion while it’s still small. Observe who’s adapting well and ask how they’re doing it.

If your team doesn’t provide it, find your own safety nets. Create small circles of safety where you can be incomplete. Demonstrate by example, create this safety net for yourselves and others.

Change forces everyone into a beginner state. The only question is how long you’ll spend thinking you have passed this stage.

Some of the difficulty during those early changes came from maintaining appearances. Once I stopped pretending, I started seeing what I actually needed to rebuild.

I realized that navigating change isn’t about waiting for the ground to settle—it’s about learning to move while it’s shifting.

Part 2: What Change Actually Depletes

When the ground shifts, it’s not just about adhering to the new norm or the new context. It’s about losing the invisible infrastructure that makes work feel doable. How can I be effective? How can I keep things under control? Who can I rely on?

The bandage gets stripped away. That "beginner state" I’d learned to admit was becoming my understanding of what was depleting.

I didn’t know what success looked like anymore. I didn’t know what mistakes to avoid. I didn’t even know what people expected me to do. Each new team brought different dynamics to decode, different trust to rebuild. Existing teams had established patterns I needed to read. Newly formed teams had no patterns yet to build upon, but from what foundation?

The Three Gauges

Later, while reflecting on this article, I found a framework that gave language to what I’d been feeling. Self-Determination Theory—developed by psychologists Edward Deci and Richard Ryan—identifies three fundamental psychological needs that fuel motivation and well-being:

Autonomy: The feeling that you have choice and agency in your actions.
Competence: The feeling that you’re effective at what you do.
Relatedness: The feeling that you’re connected to others and belong.

The Self-Determination Theory

These needs are universal, but how we experience and restore them varies. For some, autonomy means voice in decisions. For others, it’s space to execute without interruption. Same gauge, different signals.

It was a lightbulb moment… suddenly I had language for what change depletes. Like knowing the light switch exists but finally seeing where it is and how it works. This was the invisible infrastructure: autonomy, competence, relatedness—and psychological safety is the soil in which they can grow.

Change doesn’t just add uncertainty. It drains these three fuels simultaneously but they don’t operate in isolation.

The Cascade

What made it harder to diagnose was how these needs interact. And this pattern isn’t unique to me.

When we change teams, relatedness shifts. We still know people in the company—there are familiar faces, colleagues we can ask for help. But the immediate network changes. The people who understood our context, who we’d built trust with over time, aren’t in the room anymore. New members don’t know us yet. We don’t know their struggles, what makes them open up, how to earn their trust. That takes time to rebuild.

Competence can rebuild faster because we’re not starting from zero—we have the broader company context, we know how to navigate systems, nowadays even AI/LLM could provide initial guidance. But understanding the new domain deeply enough to be effective? That’s slower. We need feedback from people who are still learning to trust us. We need to understand problems we haven’t seen yet. The acceleration comes from knowing who else to ask beyond our immediate team, but the depth comes from relationships that don’t exist yet.

And when competence feels shaky, autonomy shrinks. Decisions feel riskier. We defer more, check more, second-guess more. Even in areas where we technically have authority, we don’t feel the agency to use it.

The reverse is also true: rebuilding one fuel can accelerate the others.

What Restoration looks like

The shifts weren’t dramatic. They were subtle.

Competence didn’t fully restore when I mastered the domain. It was restored in fragments: when I could answer a basic question without searching. When I understood a team member’s goals well enough to see how they fit in the team, in the company. When I could recognize their struggles and offer something useful. When I could foresee more clearly the direction we should take.

Autonomy didn’t come from being given more authority. It came from finding the small spaces I could still shape. How I approached learning. How I structured conversations. How I framed problems to help the team see different perspectives—not as a manager, but as someone who could explain why we’re in the current state and where we might be headed.

Relatedness rebuilt differently each time. Sometimes through one person who became a clarity anchor. Sometimes by observing who else was navigating similar transitions. Sometimes by positioning myself as a connection point—knowing enough about the domain to be useful glue.

We don’t need to restore everything at once. We need to recognize which gauge is limiting us the most right now, and find the smallest step that moves it up.

Returning to my experience, I’d stopped waiting for stability to return. Instead, I was learning to rebuild while the ground kept shifting.

The Shift

Understanding what was depleted converts enduring the changes into specific signals I could track and pursue.

When I feel paralyzed, I can ask: Is this a competence gap (I don’t know how) or a relatedness gap (I don’t have trust)? When I feel constrained, am I actually lacking autonomy, or have I stopped noticing the choices I still have?

The analytical lens doesn’t make adaptation easier. It makes it more navigable.

Once I could name what was missing, the question shifted. Not "How do I survive this change?" but "What’s the smallest move that restores one gauge?"

That shift—from enduring to instrumenting—didn’t eliminate what we lost. But it created space for a question I hadn’t been able to ask: what if uncertainty wasn’t just a threat, but territory to explore?

Part 3: The Space between Exhaustion and Curiosity

As soon as I could identify what was depleted and take steps to restore it, something changed: the exhaustion remained, but I could see where the future was leading me, I could see possibilities.

I noticed how people around me approached their challenges. Some colleagues had impressive output. Others had sharp minds that cut through ambiguity. Some had interesting mindsets that reframed problems in a new perspective.

I became curious. Not in an abstract way—in a specific way. How did they do that? What tools did they use? What did they think about the problem?

Some dove straight into challenges, ready to solve issues immediately. Others took a more cautious approach, observing first, then acting. Everyone had different expectations shaped by their personal context—their background, their previous experiences, what they’d learned to value.

All these approaches were just different ways of navigating the same uncertain territory.

The Paradox explained

The anxiety of change and the lure of curiosity.

Psychologists call this the Uncertainty Paradox: humans exhibit both neophobia (fear of the new) and neophilia (attraction to the new) simultaneously.

Both are evolutionary. One protects us from overload, the other drives us toward growth. Both serve us. Both are valid.

My paradox makes sense now: I like change (neophilia) and am exhausted by it (neophobia).
Self-discovery helps us understand how changes affect us differently over time and that understanding matters.

The question is not which feeling we have. It’s which one we should feed at any given moment. And whether we have the capacity to make that choice.

Why Curiosity isn’t free

Curiosity sounds light. Exploration sounds adventurous.

The reality is heavier.

For example, being curious about how my team members approached their work means energy to listen—really listen, not just hear them talk. It means taking time to understand their context, their struggles, their processes, what they are saying and what they are not saying. It means emotional support through conversations, some tough, some light.

As a manager, curiosity is not just about domain knowledge. It is also about caring for people, helping them to navigate their own uncertainties and through happy or difficult moments. Guiding them toward finding their own solutions, not just giving answers.

I’m not a trained psychologist. My emotional support is limited. But I need to care. I need to help.

The cost is not just my uncertainty anymore. It’s my uncertainty plus their uncertainty plus supporting them through theirs.

Curiosity about people is different from curiosity about systems. It costs more.

The Boundaries that make it possible

I learned something else: we cannot do everything.

We don’t have time for everything. We all have our limitations and our learning edges. We need to acknowledge those limits and work with them, not pretend they don’t exist.

Sometimes another manager or team member is better suited to help someone than we are. Therefore, we need our support network. It’s important to know who to ask and how to handle the situation.

We need to set these boundaries to sustain our work and be effective. Remember, we have three gauges to act on.

Exhaustion doesn’t disappear when we get curious. It stays. And sometimes, that’s useful—it keeps us honest about our limits.

The shift is not about being tireless. It’s more about being curious within constraints, not despite them. Choosing which uncertainties to explore and which to defer. Which conversations to have and which to delegate.

What stays

I’m still afraid when change comes. The fear, the resulting exhaustion is still there, just as present as before. Over time, I’m learning how to tame it because curiosity has traction now.

The question changed. Not "What will I lose?" but "What could I learn from them?" Not "How will I endure this?" but "How could I navigate around it?"

I started observing patterns. Patterns in how people adapted—their different approaches, their personal contexts, the choices they made when facing uncertainties.

And those patterns announced something bigger: patterns in the organization’s design.

Part 4: The Organization as a Fluid System

A new reorganization happened.

A question started forming: Why does this keep happening? Why do we keep changing the organization?

The Pattern beneath the Disruption

I’d been treating each organization change as isolated chaos. But what if this wasn’t random chaos, but a recognizable pattern?

In Dynamic Reteaming, Heidi Helfand’s research describes it as intentional, routine changes to team composition—not reactive scrambling, but deliberate organizational design. Teams form, merge, split, and switch to meet evolving product and system needs.

I was viewing teams as static groups that were being broken and recomposed. In reality, I was experiencing a lifecycle.

The Dynamic Reteaming Ecocycle

Teams can get stuck and might fall into the Poverty Trap, where a lack of resources and support prevents them from growing or might be in the Rigidity Trap, doing the same routines even as the ground shifts around them.

I could see the logic behind the chaos in glimpses. Leadership was navigating its own uncertainties—market shifts, competitive pressure, technological disruption. They were making the best decisions at the time, balancing competing needs: customer value, organizational learning, individual sustainability.

What is depleted

When I formed a new team—mixing familiar faces with people from another domain—relatedness and competence hit hardest.

I had to rebuild trust. New members didn’t know my context. I didn’t know theirs. The shortcuts we’d developed in previous teams—the unspoken understanding, the "I know what you mean" moments—were gone. We were starting from scratch in how we communicated, how we made decisions, who we could rely on.

Competence took longer. The new domain had its own vocabulary, its own problems, its own standards. I couldn’t lean completely on previous expertise. I was learning again, but this time with the added weight of supporting team members who were also learning, also rebuilding.

Then, small wins started appearing. The newly formed team was growing alongside me. Our gauges were refilling, slowly.

The research on Dynamic Reteaming suggests that we can only make the best of organizational redesigns when the environment provides psychological safety and recovery time. Without those, even well-intentioned change can become depleting rather than developing.

The recovery time wouldn’t always be perfect.

Understanding that we needed to change to adapt to business direction gave me a lens. I could see the "why" behind the disruption. I stopped personalizing it.

But understanding didn’t reduce my exhaustion. It just made it legible.

A Question that remains

Dynamic Reteaming is a strategy, not a universal good. It assumes baseline conditions: psychological safety so people can admit what they don’t know, and recovery cycles, so exhaustion doesn’t compound into harm.

When those conditions exist, the system works. Individuals build change fluency—the portable skill of adapting quickly. Organizations gain resilience. Knowledge flows.

When those conditions don’t exist, there’s a gap between organizational intent and individual experience. What leadership sees as building adaptability, individuals may experience as meaningless change.

The system’s intent matters, but so does its execution. Understanding the pattern gives me language to assess: Is this a navigation challenge or a sustainability problem?

I haven’t fully answered that yet. But I’m trying now, asking different questions.

Conclusion: Learning to Reinvent

I still feel both. The exhaustion and the curiosity. But something has shifted.
I’m curious about what the next change will bring.

I no longer wait for the ground to settle. I’ve learned to ask the questions that make me feel like a beginner. I may identify what needs restoring first by observing who’s navigating well and learning from their patterns. I can move while everything is still shifting.

Change fluency strengthens; the transition time is faster. Not because it got easier—because I learned what questions to ask.

Looking back: If nothing is permanent, then the most valuable thing I can build isn’t expertise in any single domain. It’s the ability to adapt. To recognize patterns. To restore what matters. To explore what’s new.

In an unstable world where the ground keeps shifting, permanent adaptability is more valuable than permanent expertise.

The paradox remains. I still like change. I’m still exhausted by it. But I’ve learned they’re not contradictory—they’re two sides of the same journey.

The exhaustion is my tuition; the curiosity is my compass.

What will you discover in your next transition?

Tomorrow’s article will be by otter.

Finally, Mercari Japan in English! Our Road to Cross-Platform i18n

Mon, 08 Dec 2025 10:00:40 GMT

This post is for Day 8 of Merpay & Mercoin Advent Calendar 2025.

For more than a decade, Mercari’s Marketplace has been a Japanese service – sometimes to the annoyance of our many, many employees who don’t speak Japanese. But as of late November, we’ve finally shipped English UI support for our main flea market app – across iOS, Android, and web. Huzzah! I’m fenomas, a tech lead for our website, and today I’d like to share a look behind the scenes at how it worked, why it took so long, and what comes next. (There’s also a good bilingual pun near the end.)

Getting from 1 to 2 (locales)

Normally, the biggest and hairiest part of an i18n project is when you replace all your hard-coded strings with keyed resource lookups. That is, you make changes like this:

-   <h1>My Website!</h1>
+   <h1>{ t('project.main.headline') }</h1>

…for every string, on every page, in your app or website.

In our case, there’s a twist—four or five years ago we did a “ground-up” rebuild of our apps and website, and the brilliant engineers who worked on that project (it was before I joined) had the foresight to build for i18n from day one! They used standard libraries and patterns, like i18next for web and go-i18n for backend, and our engineers have avoided hard-coded UI strings ever since.

At the source code level, we’ve been prepared to support English for several years. Why did it take so long? Looking back, there was no single reason – teams got shuffled around, and priorities changed. But one key event sticks out (for me, at least) – early on, we added a way for internal users to flip their locale over and use the English UI for their own accounts. We did this for testing and to gather feedback, but it also meant that our non-Japanese-speaking employees could switch their UI to English for their day-to-day work. And that’s a dangerous pattern – once your internal stakeholders stop feeling a pain point, it can feel less urgent and wind up being deferred in favor of other features.

So let this be a cautionary tale: it’s often necessary to add internal flags and overrides, but beware getting so used to them that you neglect to ship the feature to actual users!

Getting from localized strings to releasable strings (with AI)

Since our codebases have supported i18n for years, most of our UI strings already had English translations in the source. Until now, we had no particular process for localizing, because English support has been an unreleased internal feature. Some teams got their strings professionally translated, others used AI, and the EN strings were often written by whoever was available at the time, even if they weren’t a native speaker.

So we needed to review and fix all our English strings. At the technical level, this wasn’t such a large task – our marketplace app has around 15,000 strings, spread among several repositories. Reviewing 15K string translations would have been daunting just a few years ago, but here in the age of AI we took the following approach:

We exported all our string resources from source code into a TMS (translation management system) called Phrase Strings.
Using the TMS’s export features, we grouped all strings into unique (EN, JA) pairs.
We submitted these (in batches of 100) to an LLM, with a prompt to look for mistranslations or terms that are likely to need business or legal review.
- Tip: we found that simply asking the LLM for a list of errors didn’t work well in practice – asking it to give each pair a rating like “low/medium/high” gave better results.
We also ran simple (non-AI) scripts to flag all the strings that included the names of branded Mercari services, so we could make sure they were translated consistently.

This way, we narrowed our task down to around 1,000 strings that needed fixing or new translations. Since many of them included legal and marketing terms, we eschewed AI from this point on, and our excellent internal (human) translators took over.

The non-technical side of making strings releasable

If you take on a project like this, you may find that the technical challenge of localizing all your strings is small compared to the business challenge of getting lots of different teams to agree that their feature’s localization is definitely okay to release publicly.

We planned for this early. We expected that many other teams would have concerns about how their feature was localized, but they might not have time or resources to fully review the localizations my team wanted to release. Allaying such concerns is one of the big reasons why we decided to do a thorough AI review of all our strings, rather than just dogfooding the strings we already had and asking other teams to review the result.

The other big business concern with new localizations was QA testing. There’s a whole category of i18n-related bugs that teams rarely encounter until they support their second locale—in our case, the most common one was truncated UI strings. This can happen anywhere that the localized version of a button is longer than the source language – and our source language is Japanese, which is of course much denser than English.

But beyond small-picture bugs, like string lengths or handling plurals correctly, the big picture is that doing QA for a multi-language app has all kinds of unique challenges. Do you run all your existing tests in the new locale? Do you need new tooling in order to emulate clients with different locales? If you take on an i18n project like this, make sure your QA team has plenty of time to plan.

Other technical wrinkles

Each platform we worked on had its own little side-quests. For web, the biggest of these was routing and redirection – we decided to store the user’s preferred locale on the backend, and redirect them when they visit a route for another locale. This way, if an EN user clicks a social media link to a JA route, we redirect them back to an EN route – and vice-versa. But this means that our routing code has the potential for a redirect loop, which is something that should make every web developer think twice. Once you mix in complications like feature flags and toggles for internal dev tools, it takes a lot of care and testing to make sure users in production can never wind up in a redirection loop.

For iOS, our biggest complication was that iOS treats app locale as a system-level setting. This means that once you include resources for a new locale in your app, users with their system set to that locale may suddenly see their UI changed, without the app having a chance to ask if they wanted to switch. This isn’t really a technical challenge, but it means that when you plan to release a new locale, your UI flow will likely need to treat iOS as a special case.

Meanwhile for large services like ours, i18n isn’t just a frontend issue—a lot of localizable strings are stored in the backend as well. This is naturally true of things like error messages and notifications, but in some cases we also use server-driven UI—which means many strings that look like static UI can actually live on the backend. Since we heavily use microservices, we found our backend strings were spread out across quite a few repos—most of which supported i18n, but not all, and not all used the same i18n libraries.

Getting over the finish line

For us, the final critical step was extensive dogfooding. We did this early and often – and pro tip: getting catered snacks helps attract testers. (But not as much as when our QA engineer Alexander prepared a bunch of Android and iOS phones that already had English enabled, so dogfooding users could get started immediately.)

Dogfooding turned up a lot of fun issues. My personal favorite was that in our database of category strings, we had “Chino Pants” translated as “Chino Bread”. (If you speak Japanese this makes sense – チノパン, right?)

The other notable issue we discovered late was with how we handle mailing addresses. By default an English UI encourages users to enter their address in their display language, but some of our external logistics partners require mailing addresses to be in Japanese. For a fully global service this would typically be handled differently, but in our case we can assume that Mercari Japan users already know how to input their address in Japanese, so we just needed to make sure the UI clearly explained what inputs were required.

Then the final step for a huge i18n project is that you have to draw the line somewhere, and release some localized features while you work on the rest. Mercari has lots of services, lots of websites, and lots of dynamic features, and if we waited until everything was perfect we’d likely never release anything. So for our Phase 1, we’ve released English support for static UI only, in our main Japan marketplace apps and web, even though some related services aren’t localized yet. And moving fast this way is best for our users, after all—if you can’t read Japanese, partial localization is more useful than no localization at all.

What’s next

Our biggest next step is to translate dynamic content, like the titles and descriptions of listed items, and most crucially user comments. Getting this right won’t just be a technical problem—the process of buying and selling items on Mercari is often quite a social one, with users messaging back and forth about the state of the item, asking about discounts, and so on. The nature of a Japan-based service is that local users are likely to feel apprehensive at the thought of receiving comments in English, but if we can provide a great UX with AI-driven translations, we believe we can enable cross-language buying and selling, without changing the positive vibe we strive for in our marketplace. Look for a Phase 2 release early next year!

Wrapping up

If you set out to support i18n for a large service, I hope this article gives you some ideas of what to expect—your biggest challenges are likely to involve quality and cross‑team alignment, more than code changes and PR reviews. Manage the scope, plan for several cycles of dogfooding, and beware the pitfall of making it too easy for internal users to flip on the feature before it’s been released to end users.

And if you’re somebody who uses our apps or website in English, I hope my team’s effort made your experience a little better!

Tomorrow’s article will be by @seitau. Look forward to it!

Engineering The Semantic Layer: Principles for Data at Scale

Sat, 06 Dec 2025 11:00:34 GMT

This post is for Day 6 of Mercari Advent Calendar 2025, brought to you by sathiya from the Mercari JB Data team.

We are in an era of Data Intelligence, where we have efficient analytical datastores and advanced AI tooling, yet we are challenged in answering questions like "What was our revenue yesterday?" or "Why do different dashboards show different numbers?"

This article explores why these inconsistencies happen, how they slow down analytic operations and machine learning, and why a semantic layer is required as an essential infrastructure for data at scale.

A Fragmented Workflow

For years, most data organizations have relied on a familiar pattern:

Raw data lands in the warehouse, often including unrealistic values (e.g., placeholder prices like ¥9,999,999, draft item descriptions, or test records).
Analysts build derived tables and views to make that data usable for reporting.
Dashboards are created, limiting business users to predefined reports.
Data scientists work outside the BI layer, writing custom SQL and performing their own data cleaning before modeling.

This structure creates redundant work, inconsistent definitions, and a tangled set of transformations across teams. While these issues are usually discussed from an analytics perspective, data scientists feel the same pain – clean, reliable data matters just as much in machine learning as in BI, because poor inputs degrade model performance.

The lack of an unified semantic understanding is an issue – a "co-habitant inter-dependence” where every team relies on data but interprets it differently. When meaning is not defined consistently across the tools, the organization becomes overly dependent on their analysts and engineers to reconcile definitions, answer basic questions and reconcile conflicting dashboards.

The Challenges of a SQL-First Exploration

SQL is powerful, but relying on it as the primary interface for organization-wide data exploration introduces significant barriers:

Challenge	Description
Inconsistency	Metrics (e.g., revenue, active users), filters, and time grains are defined differently by every user.
Duplicacy and Fragility	Repetitive code (filters, joins, Common Table Expressions (CTEs) A.K.A the ‘WITH’ clause) is copy-pasted across projects, becoming difficult to govern and manage.
Data Model Knowledge Required	Users must have a thorough understanding of underlying schemas to write any meaningful, correct query.
Lack of Business Meaning	SQL focuses on how to grab data, not what that data means in a business context.

Exploration should not be gated by SQL ability. Data meaning must be standardized and accessible to all.

Semantics Layer, Simplified

A Semantic Layer is a new architectural layer that bridges the gap between raw data and end-users, serving as the single source of truth for business definitions.

Humans naturally attach meaning to symbols. When we see the word ELEPHANT, our brain instantly recalls a rich mental model – a large, majestic animal with tusks and a trunk. Data systems don’t have this intuition. They only interpret raw column names, codes, and table identifiers unless we explicitly define what they mean. This is where a Semantic Layer comes in to provide interpretation in between.

It transforms technical schemas into clear, human-understandable business concepts – for example, a table represented as tbl_usr_oo1 internally can translate to the table USERS, the field ord_amt corresponds to the Order Amount.

This translation step ensures that machines interpret data correctly, and more importantly, humans interpret it consistently across dashboards, teams, and tools. But semantics go beyond simple renaming.

A robust semantic layer embeds business rules, calculations, and governance so that the organization operates from a single shared understanding of its data. At its core, it is:

An abstraction layer over raw data
It hides the complexity of schemas, joins, and column names, exposing clean, human-readable concepts.
A single repository of truth for business logic
Every metric, filter, exclusion, and rule is defined once and reused everywhere, eliminating inconsistency.
An interpreter that exposes universal meaning to every downstream tool BI dashboards, notebooks, ML systems, and applications all consume the same definitions – no duplication, no drift.
A universal API for metrics, relationships, and concepts Tools don’t need to know how to calculate Revenue or Lifetime Value—they simply request the metric, and the semantic layer guarantees correctness.

Raw Data vs. a Semantic Layer

The following table highlights how a semantic layer fundamentally changes the way organizations work with data.

Feature	Raw Data	Semantic Layer
Nature	Technical, schema-driven, often difficult to interpret.	Logical, curated, aligned to business concepts.
Language	Tables and columns (tbl_usr_001, ord_amt).	Friendly terms (“User”, “Order Amount”).
Business Logic	Typically missing or recreated repeatedly (e.g., manually excluding cancelled orders).	Logic is embedded once (e.g., “Revenue” always excludes cancelled orders).
User Experience	Requires SQL and schema knowledge.	Drag-and-drop or natural language; no SQL required.
Risk	High chance of inconsistent metrics.	Single Source of Truth – consistent definitions everywhere.

The “What” vs. the “How”: Business Policies Encoded in the Semantic Layer

Business stakeholders define what should be counted, excluded, or labeled. The semantic layer defines how those rules are technically executed.

Aspect	Business Policy (The What)	Semantic Layer Implementation (The How)
Data Validity	“A sale counts only if the payment has been settled.”	Apply predicates such as `WHERE payment_status = 'settled'`.
Exclusions	“Exclude test orders and employee purchases.”	Hard-coded exclusion logic like `WHERE is_test_flag = 0 AND email NOT LIKE '%@company.com'`.
Calculations	“Profit = Revenue – Cost of Goods Sold – Shipping.”	Provide a reusable metric such as profit: `measure: revenue { type: sum sql: ${order_price} ;; } measure: cogs { type: sum sql: ${cost_of_goods_sold} ;; } measure: shipping { type: sum sql: ${shipping_cost} ;; } measure: profit { type: number sql: ${revenue} - ${cogs} - ${shipping} ;; description: "Profit = Revenue - COGS - Shipping (pre-tax)" }`
Null Handling	“If the region is unknown, label it ‘Unassigned’.”	Use `COALESCE(region_name, 'Unassigned')`.
Time Standards	“Our fiscal year starts on April 1.”	Provide a reusable fiscal calendar so all consumers use the same fiscal logic (assuming a fiscal year starts in April) `dimensions: date_year: type: number sql: YEAR(${TABLE}.order_date) ;; date_month: type: number sql: MONTH(${TABLE}.order_date) ;; fiscal_year: sql: CASE WHEN ${date_month} >= 4 THEN ${date_year} ELSE ${date_year} - 1 END ;; fiscal_month: sql: CASE WHEN ${date_month} >= 4 THEN ${date_month} - 3 ELSE ${date_month} + 9 END ;;`

Engineering the Principles for Scale

Building a successful semantic layer requires a deep understanding of the business’ core principles. It must act as a shared language and a foundation of trust across all teams – engineering, analytics, data science, and business operations.

To achieve this, a semantic layer must be designed to function as:

A contract and dictionary for business terms, ensuring that concepts like “Active User,” “Revenue,” or “Valid Listing” mean the same thing everywhere.
A compiler that translates business requests into optimized SQL, so users focus on intent rather than implementation.
A governance and security layer, enforcing access controls, data quality rules, and standardized definitions.
A universal metric API, exposing consistent, reusable metrics to dashboards, notebooks, ML pipelines, and applications.

Lessons from Designing the Semantic Layer at Mercari

Considering all of the above principles in mind, we decided to come up with a Semantic Layer that would abide by all of the best practices and the following were our findings.

1. One Definition of Metrics

Instead of redefining key metrics repeatedly, we coded the definition in a semantic model definition/data model configuration as:

...
measure: revenue {
  type: sum
  sql: ${order_price} ;;
}
...

All downstream models inherit the exact same definition, eliminating drift and guaranteeing consistency. When a definition changes, the core model is updated once and the change propagates automatically to every dashboard, report, or AI agent. This removes duplication and prevents conflicting logic.

When other models need to adjust or extend certain dimensions, they can do so within their inherited model definition without affecting any others. This provides flexibility while preserving a consistent foundation.

2. Logical Models over Explicit SQL Joins

Instead of the analysts or data scientists to hand-write joins repeatedly, the semantic layer represents relationships logically and lets the engine generate the optimal SQL automatically.

Traditional SQL Version

...
FROM 
    orders 
JOIN 
    customers 
ON orders.customer_id = customers.id

Equivalent Semantic Model Definition

explore: orders { 
    join: customers { 
        type: left_outer 
        sql_on: ${orders.customer_id} = ${customers.id} ;;
    }
 }

The semantic engine constructed the correct and most efficient SQL based solely on the dimensions and measures requested, fully abstracting away the complex SQL logics and table relationships. This eliminated the writing of repetitive boilerplate SQL, reduced the errors in analytical logic and ensured every downstream consumer to use consistent, and validated relationships without thinking about the underlying complexity

3. Insulation From Schema Changes

We decoupled data modeling from data querying. The Semantic layer acted as a protective buffer between warehouse changes and downstream users. When a column is renamed, a table is split, or fields are reorganized, the update is applied once in the semantic model definition. All dashboards, reports, AI agents and downstream applications continue to function without interruption. This prevented breakages, eliminated emergency fixes and ensured stability even when the schema evolves.

4. Enables Next-Generation Analytics

When meaning is consistently encoded in the semantic layer, advanced analytics become far more reliable. Natural Language Query (NLQ) systems can interpret user intent accurately, drag-and-drop BI tools generate correct and expressive queries, and external applications can access trustworthy metrics through a universal API. This foundation unlocks a new class of analytical and AI-driven capabilities without requiring every tool to understand the underlying data structures.

5. Explicit Business Meaning and Contracts

The semantic layer formalizes business definitions, turning what was once tacit, undocumented knowledge into explicit, governed contracts. It answers foundational questions such as: What defines a “customer”? What counts as an “active user”? How is Gross Merchandise Volume (GMV) calculated? By codifying these concepts, the semantic layer ensures that every team, tool, and workflow operates from the same authoritative definitions. In this way, semantic layers become the system of record for business meaning.

Conclusion: Meaning Before Measurement

The future of data is not SQL-first or dashboard-first – it is semantic-first. A strong data foundation must prioritize meaning, not mechanics.
Raw data without semantics is just storage. Data enriched with shared definitions, business logic, and consistent rules becomes trustworthy insight, operational intelligence, and scalable decision-making.

Semantics turn data into understanding, and understanding is what organizations ultimately depend on to move faster, align better, and make smarter decisions at scale.

Tomorrow’s article will be QAエンジニアがAIで日々の課題を解決した話 by Yuga Hashimoto.

A Pragmatic Approach to AI-Powered Documentation Generation

Fri, 05 Dec 2025 10:00:52 GMT

This post is for Day 5 of Merpay & Mercoin Advent Calendar 2025 , brought to you by @Fab from the Merpay Growth Platform team.

Recently, AI has taken the world of Software Development by storm. Although there are some debates about trying to apply AI everywhere incorrectly, my current team at Merpay realized that we could solve one of our documentation problem with the help of AI.

You see, my team is in charge of a system made of dozens of event pipelines with messages flowing in complex, multi-level patterns, often involving fan-in/fan-out mechanisms. We have tried for some time to create accompanying technical documentation to facilitate engineers’ onboarding when they have to work on a specific pipeline they may not know well. But over the years, the documentation efforts have trailed behind and the majority of the pipelines have become undocumented as a result.

After several tries and hours of tinkering with an AI-based approach, we are now on our way to catch-up with the backlog of documentation and considerably reduced the hours needed by engineers to write and review the documentation of each pipeline.

I feel like it would be nice to share some key moments of our journey, the approaches we took and the learnings that we made.

The “chore” of documentation

I clearly remember a quote of one of my teachers in charge of Software Development when I was at university:

“You must comment your code and extensively document your programs! When you will work in the industry, you will be at least 50% coding and 50% writing documentation.”

After I graduated, I must admit that I wrote almost no documentation at all in the first two companies I worked at, so I always felt that this statement was a bit exaggerated. Then, I started working in bigger companies with larger and more complex systems. This is when I realized that I WISHED more teams would even consider writing and maintaining documentation.

Since then, I have become quite passionate about documentation in general. Even if it takes time, I am often eager to update it, create diagrams and I am happy when someone like a new member tells me that on-boarding was smooth and facilitated by the documentation I wrote.

So I always wondered why many developers don’t really like writing/maintaining documentation. While I don’t have hard data, I’ve relied on discussions with colleagues, personal intuition, and some online conversations to identify some potential reasons:

The benefits of writing and maintaining documentation are not visible immediately, you only see them in the medium-long term.
It’s a constant effort, you have to update it as the system changes and it immediately starts to lose its value if you don’t do it diligently.
Documentation is technically “not needed”. You could release your new system or your new feature without it and it would still work the same. As a result, it is often the “variable” sacrificed when it comes to deliverability for most systems (especially for internal systems).
- The effects of documentation on productivity are not as easily quantifiable.
- Tests used to have the same reputation but over time a lot of developers and managers have realized that tests could often mean less bugs and incidents which is a more tangible metric to measure as you can often associate costs to it.
Some people have also mentioned that technical documentation specifically is not as useful as other types of documentation because code can still be used to understand the system.

The last two bullet points are interesting because it made us realize that AI could be particularly helpful for this particular documentation problem.

📌
Most people recognize that documentation is useful, but the resources that need to be invested, especially the writing, is often judged too high in most of the projects/teams.

Not all documentation in a project is made equal

There is not only one type of documentation in a project. While the following is not the unique way to categorize those types, I usually encounter:

User documentation to explain and describe to a user how to use a system/application.
Requirements / Business documentation often created by Product Managers/Owners that list all the functional requirements the system should implement.
Design / Architectural documentation often useful created before starting to implement
Technical documentation that focuses on the technical aspects of various parts of the system like API documentation, event pipelines, background jobs, implementation choices etc…

And the way you write those documents and the sources that are used can be radically different depending on the type.

For example,

User Documentation would usually focus on clear step-by-step explanations, avoid complex technical terms and a huge emphasis would be on screenshots of the various UI elements.
API documentation on the other hand would focus on having the complete list of exposed endpoints, their paths, the required parameters, the responses and errors a client can receive. We can notice that API documentation sits much closer to the code and only abstracts the implementation in a more easy-to-read format (hopefully).

📌
Types of documentation have different audiences, different sources of information and are not written the same way.

“But wait, automated documentation existed before AI”

Indeed, auto-generated documents are not new: Doxygen, Javadoc or even Swagger/OpenAPI were already a thing more than 10 years ago.

The problem is that those documentation generators:

Require metadata provided by engineers. While Swagger can populate the API endpoints names/paths, the list of parameters and maybe some of the responses, it relies heavily on the engineers’ annotations for most of the details.
Are quite rigid. Even the ones who perform some type of code analysis through various means (like reflection) to extrapolate information by themselves only apply to specific code structures or need engineers to use metaprogramming to tailor those generators to their codebase.

As someone who built a lot of APIs in the past I think Swagger/OpenAPI are fantastic tools and I am glad they existed as I could spin API documentation websites very easily.

But if we take the example of Event Pipelines, there is not enough standardization on how they are implemented to have equivalent tools.

📌
Documentation generation relying on annotations or reflection has been there for a while and it has its purpose but those tools are applicable only for specific scenarios.

LLMs to the rescue

Since ChatGPT and more generally since the advent of LLMs, there are a lot of implications of the impact of this technology and AI in general on society but in this blogpost I would like to focus on its impact on the Software Engineering world.

First it’s important to understand that LLMs are not truly intelligent. If you allow me to oversimplify a bit, LLMs are “just” huge pattern recognition machines trained on an enormous amount of data. They excel at finding certain types of links and connections in certain types of data (in software development mostly code or text coming from documents, messages, emails etc…) but they don’t really understand the concepts behind the entities, the words, the tokens it processes. There are still some classes of tasks in Software Engineering where LLMs will never be able to replace a real person and a couple of other breakthroughs will be needed.

But is it actually a problem that LLMs are not real intelligent entities? – Not necessarily in my opinion.

💬
Super strong pattern matching capabilities can actually solve a lot of real world problems.

If we compare them to the documentation generation tools mentioned earlier relying on annotations needing to follow a really strict syntax, LLMs main advantage is the ability to analyze the input in a more flexible way so it can work on a multitude of input structures and don’t require as much human written metadata.

As engineers, we should continue to do what we do when a new “fancy” tool appears that may help to solve a problem: try it, evaluate it, weigh the pros and cons, and ultimately decide how to use it.
And we should do so while acknowledging its limitations and always balancing the benefits/costs it may bring to the table.

📌
Treat LLMs and the ecosystem currently growing around them as a potential new tool in your arsenal, the same way we introduced linters, test frameworks, CI/CD pipeline, containerization etc…
Sure it’s probably the shiniest new tool we got in the past years but it is still just a tool that we have to learn to use.

How to collaborate with an AI agent

When trying to tackle a problem with the help of an AI agent, I continue to follow the same principles and framework of problem solving I have always used as if I am trying to solve the problem by myself or with another human colleague.

“All grand schemes need a plan”

First I wanted to confirm the current context of the problem, what we really wanted to do:

Have documentation for all our event pipelines. In a format that is easy-to-read, is concise, and highlights the points the most important for the engineers.
Updating the documentation over time so it is always up-to-date.

But I also wanted to check if any blocker would appear quickly in the process so I started to scribble on a notebook how our event pipelines were structured and read again the existing documentation we had written so far.

And this is when I realized that our project had a couple of interesting properties:

It is written in Golang which is a statically and strongly typed language. Every manipulated object and message have a proper type and we know what’s inside each of them.
What we wanted from our documentation was mainly technical info to give an overview of a pipeline. We weren’t interested in the details of the business rules implemented in each handler. As such we wouldn’t need the agent to ingest other references like the business specs.
- As an output, we wanted a standardized document with a common set of information for all pipelines, precise diagrams that explain the overview and some details much better than text.

⚠️ Disclaimer
As I am writing this post, new documentation dedicated solutions have emerged recently like Code Wiki from Google who claim can generate an entire wiki of documentation for the entire codebase of a project https://developers.googleblog.com/introducing-code-wiki-accelerating-your-code-understanding/
I haven’t tested it so I don’t know if it could also solve our problem. What I want to illustrate with the example of this blogpost is not necessarily the solution itself but the train of thoughts when using an AI agent in general.

Why is such preparation work important?

Strong static typing gives a lot of information to the agent, the same way IDEs have much more powerful search and refactoring abilities than with dynamically typed languages. The results may not have been of the same quality with some other dynamic languages.
LLMs have issues when their context window starts to fill so I wanted to limit as much as the source and documents the agent had to ingest even before being able to start working.
It forces us to split potential big problems into multiple sub-problems, which can help to control the size of the context window.

So I came up with a plan:

I would first spend some time generating the documentation for ONE specific event pipeline. Start from a draft, improve it. And when the quality is good enough, we would create a first template that would define the documentation structure.
I would then try to generate documentation for some additional pipelines that have slightly different structures so the agent can become aware of the possible differences and adapt the template and the generation process to be more generic.
I would then introduce automation as part of our CI pipeline to automatically update the documents as code changes.

📌
Using AI doesn’t eliminate the need to prepare what you want and why you want it. It should also not make you forget about fundamentals problem solving techniques/frameworks you may have used before.

"Well begun is half done”

The way you prompt AI agents can greatly impact the quality of the results you will get. If you try to ask the agent to solve a big problem entirely by itself in one go, it has a lot of chances to spit out an underwhelming solution that you will have to patch and fix due to too many assumptions, because it’s gone the wrong direction or because it is having hallucinations etc…

Here are some of the properties of the prompt I inputted with some simplified versions of parts of the prompts I used for illustrative purposes.

Give the proper context and explain the goals properly

“We have a system whose source code is in [location], we are trying to generate documentation for some part of it. This system is made of multiple messaging event pipelines and we are particularly interested in generating documentation for this [specific pipeline]. The source code files for this pipeline are notably specified [here, here and here]. Try to analyze this pipeline and show me a report of your interpretation. We will then build the documentation for it”.

If you have worked on projects with fuzzy or unclear specs/requirements from the beginning, you already know the pain of having non-well defined things to implement and solve. So it is important to define this part with a lot of accuracy.

Ask for an interactive discussion

“I would like this to be an interactive session. I want us to plan together, take time to prepare a draft and then progressively improve. You can create temporary documents if needed that I can review and give feedback on.”

This is important because instead of entirely delegating the task to the agent and only inspecting the final result, I wanted to build incrementally collaboratively.

In some way it is a bit similar to choosing an Agile methodology VS Waterfall.

When in doubt, ask

“Don’t hesitate to ask questions if some points are unclear. I prefer this over you making too many assumptions.”

Like in real life, I usually find it easier to work with people who ask questions when things are unclear rather than making wrong assumptions.

Analysis of the first outputs

The agent first analyzed the different files from the source code. It started from a couple of files I explicitly gave and it could use the typed input and output to find relevant information elsewhere in the code.

It then created several files with different purposes:

event_pipeline_template.md

This file served as a template for a typical documentation page of an event pipeline.
At the beginning it only created a list of “Sections” that would likely be included (here is a snippet):

Quick reference
Architecture Overview (Pipeline stages, Event Flow diagram)
Dependencies (Upstream and Downstream)
Data flow
Event entities (External inputs, Internal, Outputs)
Errors returned
Log Pattern
Test scenarios

This template would be modified and improved as the discussion progressed. The goal is to reuse this template for all subsequent requests to generate event pipeline documentation.

structure_proposal.md

This file was used as an explanation about all the choices the agent made to generate the pipeline template and each section. It also included sections it considered but decided to not include for now and the reasons why.

review_document.md

This file was basically a feedback form. It contained a huge questionnaire about the choices mentioned instructure_proposal.md with feedback input fields for each of them that I could fill.

I was quite impressed with these output files, the suggested template and the explanations all deserved proper reflection and didn’t seem weird.

📌
Sometimes the direction you take from the beginning can have huge impacts on where you will end. That’s why having a plan and having well thought first quality prompts can be important.

Are all paths really leading to Rome?

The agent also gave me the choice of 2 ways of continuing the process:

A short path: I would let the AI build directly a document generated from the template and the code of the event pipeline.
A longer path: I would go through the questionnaire of review_document.md to review the more structural aspects of the template first. Give feedback and continue to refine some aspects with the agent before generating the documentation.

I really appreciated the choice offered to me as they both have pros and cons.

With the short path, you can immediately perceive the good and bad points of the template with real data. You can realize that a section you thought would be useful is actually not that important. But you can also miss potential sections that are not included and would be beneficial. You can get tunnel vision and inadvertently rely only on what the AI outputs.

If you go the long path, you have to review a more extensive number of propositions and have to make as many decisions making the process more time consuming upfront but hopefully you can end up with a higher quality result.

I decided to follow the long path as I must admit I was curious about the other “ideas” the agent had and wanted to understand why it chose some sections over others.

The importance of the feedback loop

That’s when it became really interesting. As I filled the questionnaire, I started to have other ideas on how to combine sections and use diagrams for efficient information communication. It was similar to a brainstorming session. Some ideas generated by the machine helped the creation of other ideas by me, the engineer.

This phase of going through the questionnaire, writing feedback and having new ideas and developing them took a session of focused time that wasn’t short at all. But I feel this was productive and constructive time spent.

After ingesting my answers, the agent came back at me and we had a feedback loop that went on both sides for a couple of iterations. The agent didn’t hesitate to give previews of some of the results by using the pipeline it analyzed as illustrated examples

We then reached the point where the first draft of the documentation was created and I think it turned out really well for a first draft. After a couple of adjustments the document was already on a quality level higher than the previously handcrafted documents we had.

As an experiment I duplicated the context and rewinded to the point I tried the short path. Of course I will never know for sure but I feel the final document using this approach would have lacked some of the nice things I “brainstormed” by filling this questionnaire, because I was exposed to fewer propositions/suggestions and the “reasoning”. The documentation the agent generated through the short path was not bad but I could see a lot of differences with the first one I got with the longer path.

📌
Whether it is with an AI agent or not. Spending a bit more constructive time has an impact on the quality when trying to solve a problem.

Keeping human reviews

Once I got my first draft I was happy with, I created a PR and asked review from my colleagues,

In our team we review not only code but also documentation. I would say that the current trend at Mercari is to even be more strict during reviews for Agent generated code.

But maybe some of you may wonder:

Wouldn’t mandatory reviews for documentation be a bottleneck in the process? Wouldn’t it be possible to have another AI reviewing the updates?
And wouldn’t forcing human reviews increase the likelihood of the documentation never being updated nor merged?

I think those are valid concerns but I also think it depends on how the team/organization values documentation.

I am still convinced that the part of the documentation process that causes the most friction is the writing and the necessity to update it as code changes. And AI can help with that.

But if you are in a situation where even developers don’t want to review documentation updates, maybe it is a sign that this particular documentation may not be needed in the first place.

I think it is also possible for teams to decide to automate everything from end to end without reviews (not all teams review changes even written by human engineers after all) but it has to be a conscious choice made by the entire team as it may be more sensitive to mistakes and inaccuracies produced by the agent.

That’s also why, in a way, we are okay with starting to work on generating specific parts of the documentation the team feels are useful and not necessarily trying to generate the whole documentation of the project.

📌
Ultimately, we must not forget that our goal here was to create documentation targeted at the engineers who work on our system. We don’t want to generate documentation just for the sake of generating it.

Some extra takeaways

I asked the AI agent to create a prompt template in order to generate the documentation for other pipelines.
- LLMs are usually not deterministic and even with the same prompt, the output will be slightly different each time.
- Still you ideally want to use the combination of a standardized prompt and a standardized documentation template to increase the stability of the output.
- A good reusable prompt also packs enough context information so the agent has everything it needs to do the task as efficiently as possible because LLMs do not have intrinsic memory.
I integrated an AI agent step in our CI pipeline with a custom prompt that would look for changes in event pipelines source code files and update the documentation if needed by using the exact same prompt template and documentation template obtained previously to generate/update the documentation and it would then create a separate PR with the documentation changes.
I had to be cost conscious! Something that is not well communicated with all the AI hype is the cost of AI agents execution which is not free. In our case, the agent triggered to check and update documentation was costing us a whooping 0.5 USD per execution. I quickly had to change the execution policy to inspect only merges on certain target branches instead of checking all commits pushed otherwise it would have cost our team several hundred/thousands USD per month. For all AI activities, calculate the cost relative to the benefits, like any other tool.
We generated the rest of the documentation over weeks instead of generating the documentation for all the pipelines in one go to allow engineers to review without being overwhelmed by the sheer quantity of documents.

And to wrap-up

Was it a revolutionary project? No, but all projects (being AI assisted or not) don’t necessarily have to be. We had a particular documentation problem and in that case AI helped us fix it.

So what did we learn from all of this?

Documentation is something that highly depends on each team’s practices but it seems that still a lot of them recognize the benefit of documenting at least certain parts of their systems.
- Writing documentation seems to be the most painful part of the process.
- You don’t need to document everything, focus on the parts that would benefit the most.
- Don’t hesitate to ask every time an engineer onboards to your team/system if they think some additional documentation could have helped and which part.
AI can help with documentation in general but technical documentation close to the code especially seems to offer good potential.
Continue to follow some of the same principles as with other human engineers:
- Have a plan (even if simple in the beginning).
- Clearly define the WHAT and WHY. It’s probably not a real problem that deserves attention if you cannot define those.
- Split big problems into smaller more manageable sub-problems.
- Introduce a feedback loop when possible, AI agents still give higher quality results if you support them properly. This can also help you to find new ideas.
Keep some degree of human review depending on the situation and the criticality of the task, especially for output targeted at humans.

AI can be fantastic sometimes but just because we can now automate and create a lot of content easily with AI agents doesn’t mean that we necessarily have to. Continue to be pragmatic, treat AI as a tool and as any tool, learn how and where to use it. This approach will give you the best results.

Tomorrow’s article will be by @Sakabe. Please look forward to it!

Enhancing Developer Experience through Mercari’s Unified Platform Interface

Thu, 04 Dec 2025 11:00:27 GMT

This post is for Day 4 of Mercari Advent Calendar 2025, brought to you by @whhygee from the Mercari Enablement Tools & Interfaces team.

At Mercari, the Enablement Tools and Interfaces team—responsible for developer experience and CI/CD—is building a service called Single Front Door (SFD). SFD provides developers with a single, unified interface to our Platform and helps us scale GitOps across thousands of components. We do this by combining our widely used internal command-line tool with various external services—such as Google Cloud services—to streamline the developer experience, enforce governance, maintain consistency, and make large-scale GitOps manageable.

We recently introduced a cloud-hosted Model Context Protocol (MCP) server as an additional interface, allowing developers to access all workflows and platform capabilities directly through AI-powered tools, including their integrated development environments (IDEs).

Concept

Mercari Group’s platform has grown up for years to run hundreds of production services and over 1600+ active repositories, wherein many tools—such as infrastructure-as-code repositories, abstraction framework for infrastructure configurations and application manifests, in-house CI/CD systems and many more—have been provided to support development. However, these components were missing a centralized interface. This forced developers to understand and interact with each of them separately, which demanded making changes in at least 5 repositories and completing about a dozen steps to release new services in production. Some examples of the most common interactions with the platform include:

Commit ‘Infrastructure as Code’ for resource management
Commit Kubernetes Manifests for service configurations
Commit Protobuf definitions for intra-service communication
Setup debug environments using external tools/services
Commit edits to delete/manage cloud resources

This was reported as one of the pain points to development productivity. In response, we built “SFD” as a new unified interface for Mercari Group’s platform, so users no longer need to touch multiple tools directly when they want to perform common platform operations

Since Mercari relies heavily on GitOps, most operations involving platform tools could be performed through predefined workflows which modify files via templates. These workflows use developer credentials for making changesets on our repositories on their behalf. Users can just use SFD to trigger said workflows—either through our internal CLI tool or by directly chatting with an AI agent through their IDEs and an MCP server (example below)—which will then perform subsequent steps like making changes in configuration repositories on behalf of the users themselves.

System Design

A workflow triggered through SFD goes through the following lifecycle:

Users authenticate to SFD’s through an OAuth flow and express intent through CLI prompts or natural human language input through AI chat in their IDEs.
The user interface (CLI or IDE agent) sends a corresponding workflow request to the backend along with the OAuth token.
Backend stores workflow metadata safely, then triggers a workflow using Argo Workflows.
Argo Workflows spins up execution containers for each step of the workflow.
a. These steps are executed sequentially following a ‘Directed Acyclic Graph’ defined in the workflow definition.
Kubernetes containers for each step of the workflow execute business logic, creating changesets on GitHub or other relevant components.

Disclaimer: Logos are trademarks of their respective owners.

Challenges

Safeguarding GitOps at Scale

One of the most critical elements of a successful GitOps practice is the scope of access held by the committer. GitOps treats the Git repository as the definitive configuration and operational ledger wherein who (or what) is allowed to commit becomes just as important as what is being committed.

This means every commit has the potential to trigger real, automated changes—deployments, rollouts, environment modifications, policy shifts, and in some cases (as applicable to Mercari) full-scale infrastructure provisioning. Because of this, the credentials associated with each commit effectively serve as an execution token with potentially wide operational blast radius.

In this setup, we make sure each workflow’s changes are done by:

Using the user’s own credentials, so self-triggered workflows don’t bypass human review.
Limiting automations to only the action scopes they actually need.

The team’s solution was to start an OAuth flow using our organization’s GitHub App before triggering any workflow. This gives each user a temporary access token for the app, ensuring that workflows can only interact with the repositories and components the GitHub App is permitted to access—regardless of the user’s personal permissions—reducing the blast radius.

The token is then sent to the backend, where it is encrypted and stored in a centralized datastore. Each workflow job (running in separate container pods) retrieves this token during execution, so every changeset is created using the same credentials despite each job running in its own isolated environment.

Configuring IAM + RBAC to Provide Safe Access to External Services

Another major challenge was giving both our core backend services and our Argo Workflows job containers safe access to external systems—GCP services (Secret Manager, Datastore, KMS, Pub/Sub), GitHub, GetDX, and Slack—without embedding static credentials or granting broad permissions. This was critical because our architecture follows a zero-trust model, where each workload must prove its identity and only receives the minimum access it needs.

Argo Workflows helped us address this cleanly using GCP Identity and Access Management (IAM) and Kubernetes Role-Based Access Control (RBAC). Every workflow step runs under its own Kubernetes Service Account, which we map to a tightly scoped GCP Service Account through Workload Identity. This gives each step least-privileged access to GCP APIs (e.g., secretmanager.secretAccessor, pubsub.publisher, datastore.viewer) with no shared credentials or long-lived tokens.

Our backend services follow the same pattern: each service has its own identity, its own limited permissions, and no secrets injected into pods. RBAC controls what workloads can do inside the cluster, while IAM controls what they can do in GCP. Together, they enforce strong isolation and naturally support our zero-trust design.

Envisioned End State / Golden Path

In the end state we’re working toward, this service becomes a fully modular workflow engine, where every stage of the application lifecycle is built from reusable, well-defined building blocks. Each block represents a platform capability—service configuration, infrastructure provisioning, service mesh enablement, observability, CI/CD integration, and more—and teams can assemble them into workflows that meet their needs while still following platform standards.

Instead of writing custom automation for every service, developers simply use these prebuilt building blocks, which already package production-ready defaults: Terraform modules, Kubernetes manifests, service templates, logging/metrics pipelines, and so on.

This model is intentionally extensible. Platform teams can add new building blocks for their components whenever they want to expose their capabilities through SFD, promoting innersource by design. As the platform expands, so does the catalog of reusable steps—allowing workflows to evolve naturally while staying aligned with organizational best practices.

If you’d like to explore more great work by Mercari’s Engineering teams, be sure to check out our Engineering Portal.

Tomorrow’s article will be by @mattsuu. Merry Christmas!

Shops Monorepo Five Years Later: A Tale of Bazel and Cursor

Wed, 03 Dec 2025 11:00:31 GMT

This post is for Day 3 of the Mercari Advent Calendar 2025.

Introduction

Hi, I’m Jazz from the Mercari Shops Enabling team. Our team handles a variety of responsibilities in Mercari Shops, ranging from backend, to observability, all the way to CI/CD. Our mission is to ensure the engineers who work on Mercari features have a great technical foundation and excellent developer experience.

Five years ago, Mercari Shops adopted a monorepo structure using Bazel on top of a microservices architecture. At the time, we believed this stack would support our early product phase, enabling fast iteration towards a usable product. Today, we believe the monorepo is still the right choice, but maintaining it has required us to address significant technical debt.

Over time, our setup became overly complex. We faced conflicting dependencies and unstable fixes that made standard tasks, like upgrading the Go version, difficult. These difficulties had their own consequences, as the usage of certain libraries, including internal Mercari standard ones, was blocked due to Bazel conflicts. Furthermore, while our frontend, backend, and protocol buffers lived in the same repository, they were effectively isolated by incompatible build systems.

In this post, I will share how we unified our build processes and resolved years of technical debt. I will also explain an unexpected benefit of this cleanup: our standardized monorepo became highly compatible with AI tools. This allowed us to onboard tools like Cursor and Claude Code quickly and see an immediate productivity boost.

If you are managing build system technical debt, considering a monorepo, or looking for practical examples of how AI integrates with large codebases, this article is for you.

A Quick Recap: Why Mercari Shops Chose a Monorepo

Back in 2021, when we were building Mercari Shops, we made a specific architectural bet. Unlike the main Mercari marketplace app, which was migrating from a monolith to microservices, Shops started as microservices from day one, which allowed a fast rate of delivery of features.

To manage the complexity of multiple services sharing code, we chose a monorepo powered by Bazel.

The Design Goals:

Single Source of Truth: Understand the entire service from one repo.
Shared Patterns: Consistency across Go (backend), Python (ML), and Protocol Buffers.
Atomic Changes: Make global changes apply to everything at once.

For the first few years, this worked well. But as the team grew and deadlines pressed, entropy set in.

You can read an in depth overview of the Mercari Shops initial architecture decisions in this (Japanese language) blog post: Mercari Shops Tech Stack.

The Drift: When a Healthy Monorepo Decays

By year 4, we were facing a significant problem: Toolchain Decay.

While the application code was healthy, the build configuration holding it together had become brittle. We saw classic symptoms:

Dependency Conflicts: We were stuck on older versions of Go because different parts of the monorepo had conflicting requirements. Updating one Bazel module often broke another.
The "Hack" Layer: Urgent fixes often turned into permanent hacks. We had custom shell scripts wrapped in Bazel rules and legacy flags that nobody fully remembered.
The "Bus Factor" in CI: There were code paths in our CI pipelines that only one or two people dared to touch. A simple task like "bump the Go version" could spiral into a multi-week drama of fighting conflicting Bazel modules.

The repository had become a "maze." New joiners faced a steep learning curve just to run tests locally, and developers were afraid to touch build files lest they break a service they didn’t own. Library dependencies stopped being updated, and the Go toolchain remained stuck on version 1.19, while the current version was already 1.24.

The Nightmare: An Unpredictable Toolchain Unfit for a Crisis

The toolchain decay started to become unmanageable once the builds became unpredictable. Due to heavy reliance on the rules_docker module, and its container_run_and_commit_layer rule, which is not a hermetic, repeatable way of building containers, the success build rate for any microservice dropped to below 50%. Bug reports on the fact that the rule was buggy went unanswered. Mercari Shops developers were forced to retrigger their builds multiple times until their change actually completed its full CI/CD cycle.

The result of this unreliability was as expected: there were a few near misses, where the incident remediation was delayed due to the need to continuously retrigger the build until it was finally completed successfully. Features were delayed because adding new dependencies caused the build to fail, without any meaningful feedback from the tooling on how to fix it.

At this point, the Bazel build system had become a serious threat.

The Renovation: Modernizing Our Toolchain

We decided that we couldn’t "move fast" (one of Mercari’s core values) if our legs were tied together by technical debt. We launched a focused initiative to clean up the repository.

1. Inventory and Mapping

Before touching anything, we had to figure out what we actually had. We scanned the repo to map the current state of the monorepo:

Which services used which language versions?
Which code was still in use, and which code was abandoned and not deployed anymore?
Where were the custom hacks hiding?

We needed to move from "it works, sometimes, if you do this" to "it works, always".

We found out that:

100% of the Python code in the monorepo was abandoned, and we didn’t need to keep it.
There were more than 120 different Github tasks configured in the monorepo, covering build, deployment, synchronization of settings, tests, report generation, and database management. We found that more than 20 of these tasks were completely abandoned, and were never executed.
There were more than 70 Go backend microservices, and 6 Typescript frontend services.
We couldn’t update the dependencies of the Go microservices, as they conflicted with the older versions of Bazel modules that we were unable to update.
Several of the Bazel modules we used were outdated, and some of them abandoned.
The custom hacks we had in our repo ranged from scripts to fix misconfigured automations where the script corrected the outcome of the automation, all the way to patches to libraries to avoid build errors.

2. Getting Bazel to default

The first challenge in the cleanup was bringing the Bazel setup back to a default mode, without scripts, hacks, and patches. We had tried to untangle the heavily hacked setup by upgrading specific modules, one at a time, but the patchwork of hacks made that impossible: changing one version broke something unrelated.

The only option we had was to rewrite the build system from scratch, using up to date versions of Bazel and its modules, so that it would build the code that we had live today, and not necessarily conform to the history that the old setup accumulated.

3. Migrating from rules_docker to rules_oci

A major part of the cleanup was ripping out rules_docker. This ruleset was effectively unmaintained and became a liability. We migrated to rules_oci, the modern standard for building container images in Bazel.

rules_oci is faster, standard-compliant, and separates the build from the container runtime. It is well maintained, and we are able to continuously update our project to the latest version of this module without running into issues. Their documentation includes a migration guide that provides meaningful advice on performing the migration, which was helpful to understand the differences between rules_docker and rules_oci, even if we were rebuilding the tooling from scratch.

Our builds became deterministic and significantly faster. We could finally use standard tools to sign and verify images. As an added benefit, we were able to switch to distroless images, which reduced the risk surface of our deployments.

4. The big PR and it release

Rebuilding the tooling from scratch had a downside: we couldn’t do a gradual update of the repo, we needed to do it in a single pull request. After three months of intense work, we finally merged the big PR. Some interesting numbers:

It had 118 commits
It changed 757 files
It added 37,570 lines and removed 25,978 lines of code

We had to review and approve it using github command line tools because the web interface froze due to the size of the pull request. While it was nerve wrecking to merge such a large change, the migration was a success, and smaller issues, such as adjusting the name of containers, were easily solved now that we had a tooling that gave us meaningful feedback.

The Unexpected Finding: AI-Readability

This is where the story gets interesting.

Around the time we finished the cleanup, AI coding tools like Cursor and Claude Code started becoming mainstream. We, like many teams, tried them out. The difference in their performance before and after the cleanup was night and day.

Why "Standard" Code is AI Fuel

Before the cleanup, when we asked an AI agent to "add a new endpoint," it would fail. It couldn’t understand our custom hacks, our weird directory structures, or why rules_docker was behaving strangely. The AI would hallucinate standard Bazel rules that didn’t exist in our custom setup.

After the cleanup, the repo was "boring"—and AI loves boring.

Because we were now using vanilla rules_oci and standard Go rules:

Context Discovery: The AI tools could traverse the project and accurately map the dependency graph, and the relation between different parts of the project
Correct Code Generation: When Cursor generated code, it used the standard patterns for both the feature and the build system, and for the first time, those patterns actually worked in our repo. This predictability increased our engineers’ confidence in using AI tools.

Success Stories: Humans + AI

1. The Junior DevOps Engineer

A new team member joined with strong application skills but very little experience with Bazel or CI/CD pipelines. In the "old" world, assigning them a CI task would have been a recipe for frustration. They would have to spend days learning the basics of Bazel, understanding how the scripts and hacks influenced the outcomes of the build system, and engage in a long cycle of trial and error to complete their tasks.

Instead, they used Cursor. They asked: "My service x has a race condition. How can I enable the golang race detector in my Bazel build?"

Because the repo used standard implementations, the AI quickly provided him with the correct way of enabling the race detector. Then they ran the build with the detector enabled, found out the issue, and relied on Cursor to find a solution for the issue.

Since the build system was now reliable and repeatable, they were able to quickly validate the solution with confidence.

2. AI as a Discovery Tool

We found that AI wasn’t just for writing code; it was for finding it. For example, we can ask "Find the flow from API endpoint Y down to the database writes." with a high rate of success. Mapping out complex business rules became a matter of requesting an AI agent to build a UML flow diagram.

With a cleaned-up architecture, these queries returned useful, high-coverage answers. We could use AI to sketch large refactors across the monorepo, then execute them step by step.

Lessons Learned

The most important lesson we learned is that a build system isn’t "set and forget." It requires ownership. If you don’t schedule regular hygiene work for your infra, you will eventually pay 10x the cost in slow upgrades and developer frustration.

Trying to "throw AI" at a messy, hacked-together build system just amplifies the mess. Clean code is essential for AI maintainability.

By paying down our technical debt and embracing standard tools, we didn’t just fix our build times—we opened the door for our team to build faster and smarter with AI. The monorepo is no longer a burden; it is once again a competitive advantage.

Tomorrow’s article will be by whygee about enhancing DX through Mercari’s Unified Platform Interface. Stay tuned!

LLM Key Server: Providing Secure and Convenient Access to Internal LLM APIs

Tue, 02 Dec 2025 11:00:25 GMT

This post is for Day 2 of Mercari Advent Calendar 2025, brought to you by @hi120ki from the Mercari AI Security team.

At Mercari, various initiatives are underway to expand the use of AI and LLMs within the company. To support these efforts, the AI Security team developed the LLM Key Server, a service designed to provide secure yet convenient access to LLM APIs.

This system replaces the previous manual process where administrators would register users upon receiving LLM API access requests. Now, users can obtain temporary API keys through their internal accounts without submitting access requests.

Additionally, we provide common templates for using LLM APIs in GitHub Actions and Google Apps Script, facilitating LLM adoption in local environments and across multiple services such as CI, cloud platforms, and no-code tools.

This article explains the security challenges of LLM APIs, improvements to our processes, the architecture of the LLM Key Server, and key implementation points.

Security Challenges in LLM APIs

Various LLM models are currently offered by different providers, and at Mercari, we leverage multiple LLM models based on task requirements and employee preferences. However, the APIs that provide access to these models typically require API keys.

API keys used to access major LLM vendor APIs typically have no expiration date. If a key is leaked and the breach goes undetected, organizations face the risk of prolonged information leakage and financial losses. Furthermore, the current surge in AI and LLM adoption has led to the proliferation of API keys, raising concerns about unclear management practices. Managing users, teams, and permissions across multiple LLM providers adds additional complexity. This complexity makes regular access audits difficult to conduct.

The most secure and recommended approach we advocate internally is to access LLM APIs through Google Cloud or Azure using Workload Identity and cross-cloud federation, eliminating the need for API keys. However, the complexity of such configurations, combined with the fact that many external AI and LLM products are released without supporting these methods, necessitated an alternative approach, particularly when evaluating various LLM tools.

An additional requirement was to ensure both convenience and security. Overly cumbersome security policies can paradoxically encourage users to bypass them, so we needed to pursue both safety and usability.

Providing Secure and Convenient LLM API Access

To provide secure and convenient access to LLM APIs, we decided to leverage the open source project LiteLLM, which enables access to multiple models through a single unified API, along with the OpenID Connect (OIDC) ID token issuance capabilities of Google Workspace and Google Cloud.

LiteLLM is an open source solution that makes LLM models from various providers accessible through a single API. Beyond basic LLM API calls, it also supports coding agent tools such as Claude Code.

The OIDC ID token issuance feature allows us to obtain ID tokens signed by Google by leveraging Google OAuth or service account permissions, enabling reliable user identity verification.

At Mercari, we operate a Token Server that enables access to GitHub from Google Cloud using short-lived credentials. The LLM Key Server builds upon this architecture, extending it to support LLM access.

LLM Key Server Architecture

The LLM Key Server authentication flow works as follows.

The LLM Key Server authentication flow

First, users or workloads who need LLM access for Claude Code or other applications obtain an OIDC ID token from Google APIs to prove their identity. This can be done through Google Workspace account authentication or service account authentication from the Compute metadata server.

Next, when the OIDC ID token is sent to the LLM Key Server, the server verifies the token signature and issues a temporary API key for accessing LiteLLM based on the information in the token. This API key has a short expiration period, allowing users to access various LLM models through LiteLLM.

For local environments using Google Workspace account authentication, we provide an internal CLI tool that initiates the OAuth authorization flow with a single command, handling the entire process from obtaining the OIDC ID token to retrieving the LLM API key.

When using service accounts, API keys expire after one hour. However, recognizing that cloud applications using LLMs may run for extended periods, we provide an automatic key renewal mechanism. This is implemented as a Go library that automatically renews keys, enabling continuous LLM API usage.

This approach leverages Google Workspace and Google Cloud service account authentication to provide secure LLM API access, while time-limited keys reduce information leakage risks and automatic renewal libraries ensure convenience.

Expanding LLM Key Server Usage Scenarios

The LLM Key Server is designed for use not only in local environments and cloud applications, but also across various internal tools and services. We specifically support the following two usage scenarios.

GitHub Actions

We provide a common template for using LLM APIs in GitHub Actions. GitHub provides OIDC ID tokens that can be used to obtain LLM API keys from the LLM Key Server, enabling access to various LLM models through LiteLLM. This has accelerated LLM adoption in CI/CD pipelines, including automated code reviews using Claude Code.

- name: Get LiteLLM Key
  id: litellm
  uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
  with:
    script: |
      const oidc_request_token = process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN;
      const oidc_request_url = process.env.ACTIONS_ID_TOKEN_REQUEST_URL;
      const oidc_resp = await fetch(`${oidc_request_url}&audience=https://key-server.example.com`, {
        headers: {Authorization: `bearer ${oidc_request_token}`},
      });
      const oidc_token = (await oidc_resp.json()).value;
      if (!oidc_token) {
        core.setFailed('Failed to retrieve OIDC token from GitHub Actions');
      }

      const res = await fetch('https://key-server.example.com/llm-key', {
        method: 'GET',
        headers: {
          'Authorization': `Bearer ${oidc_token}`,
          'Content-Type': 'application/json',
        }
      });
      if (res.status !== 200) {
        core.setFailed(`LiteLLM API Error: HTTP ${res.status}`);
      }
      const body = await res.json();
      core.setSecret(body.key);
      core.setOutput('token', body.key);

This template allows developers to securely use LLM APIs in CI/CD pipelines without directly managing API keys.

Google Apps Script

We also provide a common template for using LLM APIs in Google Apps Script. In Google Apps Script, we use OAuth scope configuration to authenticate users and obtain OIDC ID tokens.

At this point, we configure the Google Apps Script settings by opening the script editor, enabling the appsscript.json file from the settings page, and adding the necessary OAuth scopes.

  "oauthScopes": [
    "openid",
    "https://www.googleapis.com/auth/userinfo.email",
    "https://www.googleapis.com/auth/script.external_request"
  ],

With this configuration, you can obtain the OIDC ID token, retrieve the LLM API key from the LLM Key Server, and access various LLM models through LiteLLM using the following code.

function getLLMToken() {
  try {
    const cache = CacheService.getUserCache();
    const cacheKey = "llm_token";
    const cachedToken = cache.get(cacheKey);
    if (cachedToken) {
      return cachedToken;
    }
    console.log("[+] Fetching new LLM token");
    const token = ScriptApp.getIdentityToken();
    const options = {
      method: "GET",
      headers: {
        Authorization: "Bearer " + token,
      },
    };
    const response = UrlFetchApp.fetch(
      "https://key-server.example.com/llm-key",
      options,
    );
    const statusCode = response.getResponseCode();
    if (statusCode !== 200) {
      throw new Error(
        `HTTP request failed with status ${statusCode}: ${response.getContentText()}`,
      );
    }
    const responseText = response.getContentText();
    const responseData = JSON.parse(responseText);
    if (!responseData.key) {
      throw new Error("Key not found in response");
    }
    cache.put(cacheKey, responseData.key, 50 * 60); // Cache for 50 minutes
    return responseData.key;
  } catch (e) {
    console.error("Error getting LLM token: " + e.toString());
    return null;
  }
}

When validating OIDC ID tokens, we verify the user’s email address as well as the Google Cloud project backing the Apps Script is located within the organization’s system-gsuite/apps-script folder in Google Cloud. This ensures that only requests from trusted scripts are allowed.

This approach eliminates the need to store LLM API keys in plaintext within no-code tools, enabling secure LLM API usage.

This mechanism has accelerated LLM adoption within the company for use cases such as summarizing and translating internal documents.

Conclusion

We have developed and deployed the LLM Key Server as a core component to solve the problem of authenticating to LLM APIs for several common types of workload, providing a solution that is as easy to use as static API keys to both developers and non-developers. We believe that the best solution to support safe AI and LLM utilization is through solutions that are both secure and easy-to-use.

If you are interested in AI and LLM adoption or security initiatives like these at Mercari, please visit our careers page.

Tomorrow’s article will be by @Jazz. Look forward to it!

Websocket XSS vulnerability discovery: My security journey at Mercari

Mon, 01 Dec 2025 11:00:02 GMT

This post is for Day 1 of the Mercari Advent Calendar 2025, brought to you by @philolo1 from the Mercari Help Center team.

Introduction

At Mercari Engineering, we focus on learning and using cutting-edge technologies like AI-assisted development tools, and we value fundamentals like security.

In this post, I will share how I rediscovered my passion for security through Mercari’s Security Champion Program and how that learning helped me detect a Cross-Site Scripting (XSS) vulnerability in a 3rd party WebSocket integration during development before it ever reached production.

Phase 1: Rediscovering Security Fundamentals

Although I learned about security during my university days in Germany, I hadn’t practiced it for years. After joining Mercari, I learned about Mercari’s Security Champion Program and saw a great opportunity to brush up my security knowledge and apply it in a real-world environment.
One thing I love about my team at Mercari is the ability to spend part of my time outside the main product development tasks within our team. The Security program fit in perfectly: through online learning and live sessions, I studied with engineers from other teams and dove into topics like web security, mobile security, and even prompt injection.
The concept that stuck most was threat modeling: Threat modeling is a technique to sit together with a group of people and get into the mindset of a hacker to ask “How could i attack the system?”. After collecting various ideas about potential threats, we can estimate their likelihood and potential impact.

Phase 2: Spotting Potential Issues

By applying this mindset as a hacker to my own projects I was able to discover a potential issue within the Help-Center Chat system that was under development during that time. Since Help Center deals with sensitive customer data, security is very important.

To better understand the issue, let me first explain the Chat system. The Chat system consists of three main components: the Chat frontend interface that is used by the customer, the Google Contact Center AI Platform Backend and the Contact Center dashboard.

To make development of connecting the frontend with the backend easy, Google CCaaS (Contact Center as a Service) partner with the company Ujet to provide SDKs: The Web SDK, and the Headless Web SDK. While the Web SDK allows for simple integration with already provided UI / UX, the Headless SDK allows for more customization.

When a customer sends a message in the chat, that message will be sent through the Google Platform to the Customer support agent and displays the message in the customer support dashboard.

While implementing support for clickable links, I realized that when sending a message that includes a html tag through the Headless SDK, the messages were rendered as raw HTML on the agent dashboard rather than being escaped. I then also tried to display a button using the <button> html tag and successfully rendered it in the virtual agent dashboard.

Interestingly, the Web SDK automatically escaped HTML content correctly, but the Headless SDK did not. That inconsistency made me suspicious: this might not just be a harmless display issue—it could be a potential XSS vulnerability.

Phase 3: Discovering and Reproducing the Vulnerability

To further investigate the potential issue with the Web SDK, I used a tool called Burp Suite. Burp is a valuable security testing tool that lets you intercept any kind of application traffic such as a web app or native iOS/Android devices. The intercepted traffic can then be modified within the tool before sending it to the server. A hacker could use a similar tool to change the message to the customer and avoid frontend sanitation.

The idea is to write a message “test” that is transformed into html like this <div onmouseover="alert(document.domain)">hello</div>. When the Customer Support Agent hovers over the message and sees a popup, that means that javascript can be executed.

After investigation the message that is sent to CCaaS looks like this:

TWILSOCK V3.0 481
{"id":"[masked]","method":"message","active_grant":"ip_messaging","payload_type":"application/json;
charset=utf-8","http_request":{"host":"aim.us1.twilio.com","path":"/Client/v2/Services/
[service-id]/Conversations/[masked]/Messages","method":"POST","params":{},
"headers":{"Content-Type":"application/json; charset=utf-8",
"X-Twilio-Mutation-Id":"[masked]"}},"payload_size":69}
{"body":"{\"type\":\"text\",\"content\":\"test\"}","attributes":"{}"}

What we need to change is:

TWILSOCK V3.0 482
{"id":"[masked]","method":"message","active_grant":"ip_messaging","payload_type":"application/json;
charset=utf-8","http_request":{"host":"aim.us1.twilio.com","path":"/Client/v2/Services/[masked]/Conversations/[masked]/Messages","method":"POST","params":{},"headers":{"Content-Type":"application/json; charset=utf-8",
"X-Twilio-Mutation-Id":"[masked]"}},"payload_size":124}
{"body":"{\"type\":\"text\",
\"content\":\"<div onmouseover=\\\"alert(document.domain)\\\">hello</div>\"}","attributes":"{}"}

In addition to the message, the payload_size and the TWILSOCK header needed to be replaced as well. With Burp this is quite simple, you need to select Proxy, enter interception mode and use the match and replace feature to replace the message and payload size.

After this I was able to reproduce the issue and confirm the XSS vulnerability.

Phase 4: Reporting

I initially discovered the issue on March 21st 2025 and got familiar with Burp to reproduce the tool consistently. On March 24th, I wrote a report to Google, initially through the Google Bug hunter program. Unfortunately the response was taking some time, so I decided to directly create a support case within Google Cloud Platform on April 2nd. Then things went fast and the vulnerability was fixed on April 9th 2025: CCaaS Platform Release Notes – April 09 2025.

Conclusion

I am happy that not only was the issue resolved quickly, but also that I could apply my knowledge to prevent potential leaks of private information from real customer interactions.

This experience reminded me how important it is to stay curious. With the increased use of AI generated code and tools, it is even more important now than ever before to deeply inspect every piece of code.

I hope this article encourages you to learn more about security and security tools like Burp Suite! Maybe If you are lucky enough you can earn a reward though programs like Google’s Bug Hunter!

Thank you for reading this blog post and I hope you have learned something new about security.

Tomorrow’s article will be by @hi120ki about LLM Key Servers. Stay tuned!

Mercari Advent Calendar 2025 is coming up!

Wed, 26 Nov 2025 11:00:47 GMT

Hello! This is yasu_shiwaku from the Mercari Engineering Office.

We have our annual Advent Calendar blogathon event in December every year and we’ll be hosting it again this year!

We have both Mercari and Merpay/Mercoin Advent Calendar at the same time, so please check out Merpay/Mercoin side as well.

▶Merpay & Mercoin Advent Calendar 2025

We’ll be sharing our knowledge of the technologies used by our engineers at Mercari group. We hope this Advent Calendar will help you to enjoy the days leading up to Christmas.

Advent Calendars 2024

Publishing schedule

This is a collection of links to each article. I recommend bookmarking this page for the prompt update, and it will be very useful if you want to check it out at a later date.

Date	Theme / Title	Author
12/1	Websocket XSS vulnerability discovery: My security journey at Mercari	@philolo1
12/2	LLM Key Server: Providing Secure and Convenient Access to Internal LLM APIs	@Hiroki Akamatsu
12/3	Shops Monorepo Five Years Later: A Tale of Bazel and Cursor	@Jazz
12/4	Enhancing DX through Mercari’s Unified Platform Interface	@whygee
12/5	メルカリハロ Web フロントエンドの1年間の改善と学び	@mattsuu
12/6	Engineering The Semantic Layer: Principles for Data at Scale	@sathiya
12/7	QAエンジニアがAIで日々の課題を解決した話	@yuga
12/8	Navigating Change: Learning to Reinvent in an Unstable World	@Antony Chane-Hive
12/9	Search Results Quality Monitoring with LLMs	@otter
12/10	LiveContactToolにおける機微情報の取り扱い~CloudDLPを使ったマスキング	@sters
12/11	OpenID Connect Core 1.0 の Claims パラメーターの利用	@kgoro
12/12	Adsシステムの急成長を支える技術：信頼性と収益性を取り戻した「PJ-MARP」の全貌	@tokku
12/13	メルカリが、AI時代にナレッジマネジメントに投資したわけ	@t-hiroi
12/14	メルカリAdsが広告を届けるまでの話	@yanap
12/15	TiDB Resource Groupでワークロードを制御する	@ogataka50
12/16	The Cost of Speed: A Battle against Cost, Debt, and Diverging Systems	@Sneha
12/17	Building a Learning Culture with DevDojo	@mariz
12/18	Capturing Network Packets in Kubernetes	@mshibuya
12/19	AI-Native 開発を加速する AWS Kiro の導入と、Okta を活用したアカウント管理の自動化	@amenbo & @siroken3
12/19	メルカリ内部の Dynamic Client Registration 活用事例	@task
12/20	PR駆動の変更、CI/CDでOS設定を自動反映 — Terraformで実現するJamf ProのIaC＋GitOps基盤	@yu
12/21	Non-AI tasks in the AI task force：AIツール開発の現場でこそ必要な「AI以外の」技術選定	@akkie
12/22	Tales of OIDC & OAuth Security: What It Takes to Trust a Token	@Kahla
12/23	When Speed Wasn’t About Coding Faster: Our Journey to ‘One Person One Release’	@Sneha & @Yu
12/24	「AIが学習しやすいナレッジ基盤」メルカリが全社で導入したNotion Architecture ver1.0	@kiko & aisaka
12/25	AI-Nativeという選択ー正解のない時代に、メルカリが選んだ指針	@kimuras

Please bookmark this article and check it out when you want to read it so you can be aware of article publication notifications!

We’re looking forward to bringing you some interesting technology stories in the last month of 2025! I hope you’re looking forward to the Advent Calendar!

Building Tooling for Global Customer Support Operations

Tue, 25 Nov 2025 12:00:04 GMT

Hello, this is @waiting.lau and I’m a member of the Cross Border (XB) Operations (Ops) Engineering team.

Introduction: Turning the Hidden Half into a First-Class Product

When we build a product for millions of users, we often focus on the customer-facing experience: the slick UI, the smooth checkout flow, and the powerful search. But behind every great product is another critical component: the "hidden half". These are the internal tools that empower our Customer Service (CS) and Trust & Safety (TnS) teams to support users and ensure a secure marketplace.
For the new Mercari Global service, we faced a fundamental question: As we build a new global platform from scratch, how do we treat these essential internal operations as a first-class part of the product itself and not as an afterthought?
This article explores our journey to answering that question, detailing the pragmatic, phased approach we took: leveraging Mercari’s mature Japan assets for a rapid launch, while simultaneously building a new, future-proof foundation for our technology and our teams.

Learning to Decouple, Not Discard

To understand our approach, it helps to know what a complete CS operation entails. A few key components must work together:

A Help Center for user self-service.
A Contact Tool (or ticketing system) for agents to manage incoming inquiries.
An Operation Tool for CS agents to access data and perform actions (like order cancellations).
An Authorization (Authz) system to control permissions.

Mercari’s Japan business has a mature ecosystem of in-house tools covering all these areas, while the US Marketplace uses a mix of in-house solutions and third-party vendors.
We have established "Global Engineering Tenets" which guides us to have a consistent decision-making process.
To ensure consistent decision-making across this complex landscape, we followed the "Global Engineering Tenets" established for the entire Global Platform project, which were featured in a previous article.
To honor our tenet to "Learn and unlearn from past experience", we first analyzed this mature ecosystem. The initial thought was to extend all the existing tools in the Japan business. But this led us to a crucial realization, guided by another tenet: to "Keep each country’s business isolated". To achieve the velocity needed for global expansion, we had to decouple from the established JP infrastructure, not to discard its strengths, but to avoid dependencies that could slow down future rollouts.
This analysis led us to our pragmatic, hybrid strategy, which was defined by a clear distinction between what to reuse and what to build.
We chose to reuse the Help Center and Contact Tool because they are mature, modern, and most importantly, already designed with multi-tenant support. They served as stable, high-level interfaces that could be adapted for global use with minimal changes.
In contrast, we decided to build the Operation Tool, namely "Global Platform Ops Tool" from scratch. The existing one for Japan business, while powerful, is deeply integrated with numerous Japan-specific backend services. This was the key issue. Attempting to deploy the tool in a new region outside of Japan would have required either migrating a large number of these dependent services or undertaking a massive decoupling effort, both of which were impractical for our timeline.
Building a new, independent tool allowed us to create a clean foundation, free from these dependencies. This gives us the autonomy to develop and deploy features for our global users quickly.

The Starting Point: A Focused Team for a Fast Launch

To execute our initial launch, we made a deliberate decision to follow a conventional model: a new dedicated engineering team responsible for the development of "Global Platform Ops Tool", referred to here as the "Ops Tool Dev Team". This was a strategic choice guided by our primary goal – speed. For a project with a tight timeline, a single, focused team with clear ownership can move much faster than a distributed model that requires extensive coordination.
This approach was a proven method for getting a new product off the ground, just as it was in the early days of the Japan business. We knew this centralized model had long-term scaling limitations, but it was the most effective way to reduce initial complexity and ensure we could deliver the essential features needed for day-one operations.
This initial workflow, while intentionally siloed, was the pragmatic choice to get us started. It was always intended to be phase one – a bridge to a more scalable and collaborative future.

Building the Foundation: The Global Platform Ops Tool Architecture

While our day-one operations relied on existing JP tools, the primary mission of our dedicated team was to build the new technical foundation in the background: the "Global Platform Ops Tool".

A New Home in the Monorepo

A monorepo is a software development strategy where the source code for many different projects is stored in a single repository. Our global platform is built on this model, and for a deeper dive into its core design, we recommend reading the previous article from our architect published earlier in this series.
With this foundation already in place, our first major architectural decision was to build the Global Platform Ops Tool from scratch within the existing monorepo. This was a strategic choice aimed at one primary goal: aggressively reducing developer friction.
To understand our reasoning, let’s first consider the multi-repository alternative. In that model, the frontend application would live in one repository and the backend modules in multiple repositories. An engineer working on a single feature would have to make changes in multiple codebases. This creates a cascade of slowdowns: they must manage separate pull requests, reviewers must track changes across multiple repositories, and simple dependency updates, like for an updated Protobuf client, become a complex task of publishing and consuming packages. This model also creates deployment dependencies, forcing teams to coordinate separate release schedules.
Placing Ops Tool in the global platform monorepo directly solves these problems. By housing both backend and frontend code together, we create a unified developer experience. An engineer can handle everything for a single feature in one codebase and one local development environment, which eliminates context-switching and simplifies dependency management. This also ensures consistent deployment, as we leverage the same modern CI/CD pipeline as the rest of the Global Platform, removing the need to coordinate separate release schedules. Finally, it gives our team full ownership. We can iterate quickly without being a "guest" in another team’s ecosystem, subject to their schedule and tooling choices.
This unified monorepo strategy defined where our code would live. The next critical challenge was defining how it would be structured, and we’ll begin with our backend architecture.

Backend Architecture: Extending a Modular Monolith for Operations

Our backend is built on the Modular Monolith architecture, which our architect detailed in a previous post in this series. For a deep dive into the core concepts of our multi-tiered design, we highly recommend reading that article first.
Our challenge wasn’t to invent a new architecture, but to adapt this powerful foundation for the specific needs of internal operations. The core question we had to answer was: "How do we add sensitive, complex operational features without compromising the integrity of the core customer-facing logic?".
Our solution involved two key extensions to this foundation. The first was a dedicated Ops BFF (Backend for Frontend), introduced exclusively for the Ops Tool. This acts as a secure gateway that completely isolates internal traffic from the customer-facing BFF. Its primary job is to handle authentication for our employees and tailor data specifically for our admin UIs.
The second extension was the use of isolated operational endpoints. To keep operational logic separate from customer-facing logic, we often create dedicated gRPC for Ops servers within a module. However, this is not a strict rule. Our guiding principle is a clean separation of concerns, applied pragmatically. For modules where operational needs are simple like a straightforward data fetch or similar logic flows, we reuse the existing customer-facing gRPC server to avoid unnecessary complexity. A separate server is only introduced when the operational logic becomes complex or requires different security considerations.

A typical workflow, such as a "Cancel Order" operation, illustrates this approach:

A request from Ops Tool UI is first handled by the Ops BFF.
The BFF calls the gRPC method served by a dedicated "gRPC for Ops Server" on the Tier 1 Order Management module, which orchestrates specific workflows for order cancellation initiated by CS agents.
This orchestrator then calls the core Tier 2 or Tier 3 domain modules, like Order and Notification, to handle the actual state changes.

This layered design ensures that operational logic is securely isolated and properly owned.
While our rule is to place business logic in its corresponding domain, this doesn’t eliminate the need for a generic module to handle cross-cutting concerns. This can be thought of as a shared toolbox for our operations teams, providing features that don’t belong to a single business domain. Its responsibilities may include:

Managing bookmarks of flagged users, products, or orders.
Handling templates for private messages and moderation actions.
Orchestrating automation for complex internal operations.

We call this the "Ops" module. The key distinction is that it doesn’t own core business logic. The Order module still defines what it means to cancel an order but the Ops module might provide the automation script that calls the Order module as part of a larger workflow.
With our backend’s logical structure defined, the next critical challenge was to secure it with a proper authorization framework.

Secure by Design: A Declarative Authorization Model

Security is a top priority. However, in a larger development project like Global Platform, this creates a significant challenge: implementing security correctly requires a solid understanding of our internal authentication and authorization systems. Asking every product engineer to implement authorization checks correctly could be error-prone.
Our guiding principle, therefore, was to abstract this complexity. We should provide a paved road that makes it easy for engineers to do the right thing by separating the what from the how. The “what” is the simple, declarative fact of what permissions are required for an endpoint, defined right alongside the API contract. The "how" is the complex logic that enforces that check, i.e. a standardized process handled by the platform, not by individual engineers writing if/else statements in every function.
To illustrate why this is so important, let’s look at the common alternative: manually checking permissions in every API handler.

// file: service.proto

// The API contract has NO permissions defined.
rpc Greet(GreetRequest) returns (GreetResponse);

// file: service.go

// The security check is hidden in the implementation,
// easy to forget and hard to find.
func (s *myService) Greet(ctx context.Context, req *GreetRequest) (*GreetResponse, error) {

    // This check is manual and disconnected from the .proto. The function is defined in a shared auth package.
    has, err := auth.HasPermission(ctx, "data:user:read") 
    if err != nil {
        return nil, status.Error(codes.Internal, "auth failed")
    }
    if !has {
        return nil, status.Error(codes.PermissionDenied, "missing permission")
    }

    // Finally, run the actual business logic...
    // ...
}

This manual approach has three flaws. First, it’s a boilerplate. Engineers must add a few lines of code to every single gRPC method handler before the business logic. Second, it’s entirely optional. It relies on every engineer remembering to add this check. If they forget, it may lead to data leak. This problem becomes worse when dealing with granular, field-level permissions. Finally, the API contract in the .proto file and its security policy in the .go file are in separate locations. Maintaining these configurations is a nightmare and makes the system difficult to audit.
Our declarative model solves all three problems. We achieve the "what" by implementing a declarative model using custom Protobuf options. Here is the sample code.

// file: proto/framework/v1/authz.proto

syntax = "proto3";
package proto.framework.v1;

import "google/protobuf/descriptor.proto";

message Authorization {
  repeated string allows = 1;
}

// Adds custom method-level options.
extend google.protobuf.MethodOptions {
  optional Authorization authz = 51003;
}

// file: proto/gateway/v1/dummy.proto:

syntax = "proto3";
package proto.gateway.v1;

import "proto/framework/v1/authz.proto";

service DummyService {
  ...
  // Greet is the RPC to greet the user.
  rpc Greet(GreetRequest) returns (GreetResponse) {
    // Product engineer must declare this option when adding new endpoints. Lint rule can be set up to detect this issue automatically.
    option (proto.framework.v1.authz) = {
      allows: ["data:user:greet"]
      };
  }
}

The definition option (proto.framework.v1.authz) is automatically enforced by a shared authorization interceptor that runs on every request to a module before it reaches the gRPC method handler. The interceptor reads the required permissions from the proto definition and validates them against the user’s permissions. If the validation fails, the interceptor immediately rejects the request, ensuring that no unauthorized business logic is ever executed.
This design removes the burden and risk of error from the developer, eliminates this complex boilerplate, and creates a single source of truth, making our services easily auditable, meaning that anyone can understand the security posture of an entire service just by reading its API contract.
This platform-level authorization enforcement is enabled by default through our configuration, as illustrated by the configuration below.

components:
  application:
    http:
      enabled: false        # Whether the HTTP server is enabled
      port: 50000           # Listening port
      middleware:
        authorization:       # Newly added authorization module
          authz_api:
            enabled: true    # Enable internal authorization
            service_endpoint:
              address: "xxxx:10001"  # Address of the authz service
              timeout: "1s"          # Fail fast (deny access by default)

The key takeaway is that our product engineers are completely abstracted from this complexity. They don’t need to know how the interceptor works or how the permission check is performed. Their only responsibility is to add the correct proto.framework.v1.authz option to their .proto file. The framework takes care of the rest, guaranteeing security is enforced by default.

This secure, modular backend provides the power, but it’s only half the story. All this logic needs to be presented to our CS and TnS agents through an intuitive user interface. That’s where our frontend architecture comes in.

A User-First Frontend: Our Architectural Choices

On the frontend, our philosophy was guided by a single question: How do we make the tool both easy for our engineers to build and intuitive for our agents to use?
To solve the "easy to build" part, we chose a familiar and modern stack by aligning with the company’s "Web Golden Path", a recommended set of frameworks and libraries. The Global Platform Ops Tool is built on Next.js, a React framework for building full-stack web applications, allowing us to leverage the latest features of React, including React Server Components, for a fast and efficient experience.
This is the same modern stack used by our customer-facing Global Platform Web Product, which also utilizes Next.js with the App Router, as detailed in a previous article in this series. This alignment was a critical decision for velocity. It ensures that any web engineer at Mercari can be productive in the Ops Tool codebase with a minimal learning curve, as the core technologies are identical.
However, our most important user isn’t the developer; it’s the CS agent. This led to a crucial, deliberate exception to the Golden Path. When it came to the UI components, we had a choice: use Design System 4.0, our company’s new, modern standard for all customer-facing products, or use the in-house admin component library that our CS agents already know from using other internal Mercari tools.
We chose the latter. This decision prioritized the user experience of our CS agents over pure technical consistency. The rationale was simple: the small cost of a developer adapting to a familiar component library is insignificant compared to the cost of having hundreds of CS agents learn a completely new interface. This pragmatic choice ensured that when our agents switched to the new global system, the tool felt instantly intuitive – even if the backend had been completely rebuilt.

The Next Challenge: Scaling Ownership

While the dedicated team model was perfect for a focused launch, we knew from experience that it didn’t scale organizationally. The core issue isn’t just about workload; it’s about the friction of context. In a siloed model, the feature team must constantly teach the ops team the product specifications, while the ops team must teach the feature team the nuances of the tool’s codebase. This constant, two-way knowledge transfer is what ultimately becomes the bottleneck, slowing everyone down.
Our vision for the next phase was to solve this and to evolve towards a co-ownership model. The principle is simple: the team that builds a client-facing feature also builds its corresponding operational components.

Our rationale here was to eliminate the handoffs and knowledge gaps entirely. By empowering feature teams to own their operational UIs, we are not just distributing work – we are building empathy. When a product engineer sees firsthand how a CS agent interacts with their feature to solve a real user’s problem, the feedback loop becomes immediate, connecting their code directly to the people using the toolhuman. It turns ‘internal tooling’ into an integral and respected part of the product experience.
This future model where operational development is a shared responsibility is only made possible by the robust and flexible technical foundation we are building today. The monorepo, the modular architecture, and the declarative security are all designed to create a "paved road" that makes it easy for any engineer to contribute effectively.

Conclusion: A Foundation for Technology and Teamwork

Our journey began by making pragmatic decisions: we leveraged existing assets with a focused, dedicated team to ensure a stable and rapid launch. This gave us the runway to build a modern, scalable technical foundation in the background.
With this foundation now in place, we are looking ahead to evolving our team structures to create a truly holistic and collaborative development culture. We believe this approach where every engineer has a stake in the operational health of their domain will ultimately lead to a better, safer, and more supportive experience for our users around the world.
Thanks for reading, and we’re excited to continue sharing our progress on this journey.

Data-fetching strategy for Mercari Global Marketplace Web App

Fri, 21 Nov 2025 04:24:32 GMT

Building a robust data-fetching architecture is crucial for modern web applications, especially in a global marketplace web app where performance, type safety, and reliability are paramount.

Hello. I’m @vb, a Web developer from Cross Border (XB) Engineering. In this article, I will share how we implemented our data-fetching strategy using Connect Protocol for making RPCs over HTTP for our web application, in order to adhere to above stated principles.

The Foundation – Why We Chose Connect Protocol

Connect Protocol

For those who are not in the know, Connect Protocol is an HTTP-based RPC (Remote Procedure Call) protocol designed to make API communication more human-readable and debuggable while maintaining compatibility with gRPC.

The protocol maintains semantic compatibility with gRPC, abstracts the transport layer so that we can choose to use either gRPC, Connect or gRPC Web protocol without having to consider the specific behaviors of each.

Please refer to Connect Protocol for more details.

Strategic Drivers for Adoption

Our decision to adopt Connect wasn’t made in isolation—it was driven by several strategic considerations:

Backend Alignment (gRPC Consistency): We chose Connect because our Backend-for-Frontend (BFF) service (api service that our web app primarily talks to) is built entirely on the gRPC protocol. Utilizing Connect-node (Connect Protocol library for Node.js) on our Next.js server allows us to make RPCs over HTTP. This consistency across the stack reduces cognitive overhead. (The strategic function and implementation of the BFF is detailed further in subsequent sections.)

Overview of typical data-flow for our application

Type Safety from Protocol Buffers: One of the most compelling advantages is automatic type generation from Protocol Buffers definitions. This eliminates the manual work of maintaining TypeScript interfaces and ensures that our frontend types are always in sync with backend contracts. We used Buf CLI to compile Protocol Buffers definitions and generate TypeScript types and other glue code.
Reduced Boilerplate: Connect handles service definition, serialization, and deserialization automatically. This means our developers can focus on business logic rather than writing repetitive data transformation code.

The Architecture – Building the Data Access Layer

System Overview and Modular Architecture

Our codebase is structured as a modular monorepo using pnpm workspaces and Turbo. This modular architecture provides clear boundaries between different layers of the application while maintaining a single source of truth. Please refer to my colleague @gary’s article, he added more details about this modular approach.

Our global web application runs on a Next.js server and utilizes this modular architecture. This structure is divided into two major sections to enforce a strict separation of concerns and avoid code duplication:

Feature Layer: This layer is responsible for all the application’s user-facing code, encompassing mostly React Server Components.
Data Access Layer (DAL): The DAL is the centralized module responsible for abstracting and handling all communication with the BFF service. Our feature modules consume the DAL to fetch the necessary data for rendering components. Please refer to the deep-dive section for more information.

DAL interacts directly with the Backend-for-Frontend (BFF) Service. The BFF acts as a crucial intermediary wrapper for the numerous underlying Backend Microservices. This abstraction layer is strategic, as it enables our web app to optimize data fetching by allowing us to make just one API call per screen to gather all the required data necessary for that specific view.

Overview of major components and their relationships in our data-fetching architecture

Deep Dive into DAL Implementation

The Data Access Layer (DAL) module serves as our essential centralized data access layer. Let’s take a deeper dive into:

The structure of the DAL is comprehensive, providing four main components that streamline data operations:

Transport Configuration: It houses the centralized transport mechanism, which is pre-configured with a series of crucial interceptors handling logging, authentication, localization, and platform identification.
Data Services: The layer is organized into individual service modules, which are logically grouped by their respective business domains (cart, item-detail, account, etc.).
Type Exports: The DAL consumes the TypeScript SDK generated by the Proto-compilation pipeline and re-exports the relevant TypeScript types, ensuring the feature layer remains type-safe. More details of this pipeline can be found in next section.
Higher-Order Functions (HOFs): The layer includes various utility functions used to wrap API calls, standardizing common cross-cutting patterns such as authentication failure handling, error handling etc.

Transport Configuration and Interceptor Pipeline

The connect-transport is configured centrally within the DAL using createConnectTransport. This configuration specifies the target URL and defines the pipeline of interceptors that every request must pass through:

// Transport configuration with interceptors
 baseUrl: process.env.BFF_API_URL,
export const transport = createConnectTransport({
 httpVersion: '2',
 interceptors: [logger, authInterceptor, ...otherInterceptors],
});

To manage cross-cutting concerns—those aspects of the request that apply universally across all service calls—we rely on a robust Interceptor Pipeline. These functions automatically execute specialized logic before or after a request is processed, ensuring consistency without requiring repeated code in every service module. Some interceptors used:

Logger Interceptor: generates a unique identifier using crypto.randomUUID() and sets it on the request header as X-Request-Id for request tracing across the microservice architecture.
Platform Interceptor: identifies the source of the request by setting the X-Platform header to web, required because the same BFF is used by iOS and Android clients too.
Auth Interceptor: reads the authentication token from cookie and sets the Authorization header as a Bearer token.
Locale Interceptor: determines the region and locale from the host and pathname, setting the X-Region-Code and Accept-Language headers for proper internationalization.

How Features Modules Imports DAL

Feature modules (React Server Components) import specific data services from the DAL, every screen makes one call to the BFF service to fetch data for that screen. Following is an example for our item-detail feature module (responsible for item-detail screen):

// In a React Server Component
import { getItemDetailScreen } from "@dal/data/item-detail.ts";

export default async function ItemDetailPage({
  params,
}: {
  params: { id: string };
}) {
  const itemData = await getItemDetailScreen({ itemId: params.id });
  return <ItemDetailComponent data={itemData} />;
}

The Proto-Compilation Pipeline

Maintaining type consistency and contract integrity is automated through our Proto-compilation pipeline. This pipeline is implemented as a GitHub action that is automatically triggered whenever our gRPC Protocol Buffers (.proto files) are updated in the repository. The automated pipeline performs two critical jobs:

It compiles the updated .proto files into a comprehensive TypeScript SDK.
It publishes this generated TypeScript SDK as an internal npm package.

This ensures that the TypeScript SDK, ready to be consumed by the DAL, always reflects the most recent backend service contracts.

Data Flow: From Client to Server to BFF and Back (Demonstration)

Let’s trace through a complete request cycle for an item detail screen. Please refer to the following sequence diagram as I go through each steps:

User Requests a Page: User navigates to a URL like https://tw.mercari.com/items/<item-id>
Next.js Server Receives Request: The ItemDetailPage component, running on the Next.js server, executes and calls the DAL to fetch the required data.
DAL Fetches Data from gRPC BFF: The DAL invokes the necessary data service, wrapped in higher-order functions.
Request Processing Pipeline: The request goes through our interceptor pipeline: Logger, Platform, Auth, and Locale. The Transport then converts the request to HTTP/2 with a binary protobuf payload.

gRPC BFF Processing: The BFF receives the request with all necessary context headers

POST <bff-service-url>/<service-name>/<method-name>
Content-Type: application/proto
X-Request-Id: <uuid>
X-Platform: <web>
Authorization: Bearer <token>
Accept-Language: <locale>
X-Region-Code: <region>

Binary Protobuf Response: The BFF returns a binary protobuf response containing the item data.
Deserialization to JavaScript Object: Connect automatically deserializes the binary response back into a readable JavaScript object.

gRPC Status Check and Error handling: Our transport checks the gRPC status in response and handles errors appropriately via our higher-order utility functions.
For example, following is a wrapper function that gracefully handles a gRPC NotFound error by throwing Next.js’s notFound() error which forces Next.js to render not-found.tsx page.

// For 404 errors in item-detail
const withNotFoundRedirect = <TArgs extends unknown[], TReturn>(
apiCall: (...args: TArgs) => Promise<TReturn>
) => {
return async (...args: TArgs): Promise<TReturn> => {
try {
  return await apiCall(...args);
} catch (error) {
  if (error instanceof ConnectError && error.code === Code.NotFound) {
    notFound(); // Next.js 404 handling
  }
  // throws other errors down the chain to be handled by other utility functions or feature components
  throw error;
}
};
};

Render component : Feature components use the data and render the component and pass it to the browser.

Benefits of Our Approach

Our data-fetching architecture, built upon gRPC with the Connect Protocol and centralized through the Data Access Layer (DAL), delivers significant advantages across developer workflow, application performance, system maintainability, and reliability.

Developer Experience: By leveraging the automated compilation pipeline, we achieve robust Type Safety, preventing runtime errors through compile-time checks. This also leads to better Auto-completion in IDEs, and the structure enforced by the DAL ensures Consistent Patterns across the application.
Performance: This RPC-based architecture delivers tangible Performance gains. Since the application runs in a Next.js Server environment, integrating this strategy directly into the Server Component flow enables effective request Caching using React’s built-in cache() function.
Maintainability: Structuring our system around the DAL significantly enhances Maintainability. We achieve Centralized Logic by placing all data access code in one dedicated module, which enforces a strong Separation of Concerns . The resulting modular design means that each layer can be tested independently, simplifying debugging and future updates.
Reliability: Finally, reliability is fundamentally improved through predictable architectural components. Our use of interceptors and higher-order functions establishes clear Error Boundaries and Predictable error handling patterns. The system is designed to support Graceful Degradation under adverse circumstances by using proper fallbacks for various error conditions.

Conclusion

Our gRPC with Connect architecture provides a robust foundation for data fetching in our Global marketplace web application. By centralizing data access in the DAL, implementing comprehensive interceptors, and using higher-order functions for common patterns, we’ve created a system that is both powerful and developer-friendly.

The combination of type safety, performance benefits, and consistent error handling makes this architecture well-suited for a complex marketplace application where reliability and user experience are critical.

As we continue to evolve our platform, this architecture provides the flexibility to add new features while maintaining the quality and consistency that our users expect.

If you enjoyed reading this article, please checkout other articles by my team in our series here.

Behind the Global Launch: Decoding the Android Engineering Strategy for Our New App

Thu, 20 Nov 2025 14:21:32 GMT

Hello! I’m Karthi, an Android engineer on the Cross Border (XB) Client Core team responsible for building foundational code for Mercari’s global apps.This article is part of the series discussing how we developed a new global service and covers android engineering strategy for global app development.

Building a product for a worldwide audience takes careful planning, especially when we can leverage a proven foundation. Our goal for the Global app is ambitious: deliver a unified, high‑performance experience at global scale. This post pulls back the curtain on the key decisions behind our approach, including the context and trade‑offs that shaped our roadmap.

The Technology Cornerstone: Why We Chose Native

The choice of tech stack had to reflect not only theoretical advantages, but also our ability to build on existing knowledge and infrastructure. We chose native development based on first‑hand experience operating multiple stacks across different Mercari apps:

Mercari Japan Marketplace: Jetpack Compose first, MVVM
Mercari US: React Native
Mercari Hallo: Flutter

For the new app, we prioritized three criteria:

Faster time to market
A small, focused team
A scalable foundation for long‑term growth

On paper, hybrid technologies like Flutter or React Native can seem like the obvious fit for faster development. In practice, the right choice depends on the organization’s existing capabilities and assets.

Mercari has experience across building both hybrid and native apps. Our largest product is the Japan Marketplace app (10M+ downloads), a Compose‑first app fully rewritten about three years ago. Its architecture has proven itself at scale, and we’ve since built a rich set of reusable libraries for authentication, client event logging, experimentation, and more, along with a robust CI/CD system.

By reusing this foundation—platform libraries, infrastructure, and a large native knowledge base—we can move faster while keeping a scalable, shared platform across apps.

On high level our native tech stack looks like below

Tradeoff: Avoiding past complexity

While cross‑platform tools offer advantages, our experience building Mercari Hallo with Flutter highlighted a major drawback for us: our existing foundations weren’t reusable. We would have needed to recreate core libraries and tooling from scratch, slowing delivery and increasing risk. To maintain speed and stability, native was the clear strategic choice for us.

Streamlining Development: Embracing the Monorepo

If reuse is a cornerstone, structure matters. We considered two obvious approaches:

A monorepo hosting apps and libraries
Multiple repositories separating apps and libraries

Each has pros and cons. A monorepo centralises code, improves consistency, and simplifies maintenance. Multiple repositories keep code isolated and can reduce dependency complexity.

We chose Monorepo, reinforced with clear boundaries and strict dependency rules, to capture monorepo’s benefits while limiting sprawl. On the top level our Monorepo structure has 2 major sub directories : product, library

Product: Contains independent directories for products like jp and global. Each product has its own app , core and feature directories and its modules.
Library: Contains all foundation modules used across products.

To ensure the structure remains clean and dependencies don’t cross boundaries, we enforce strict separation rules using the modules-graph-assert plugin. This plugin validates our dependency graph during CI and fails the build if any violations are detected. To find violations, we have two major rules for dependency isolation. 1) Product modules must remain isolated from each other (Ex: product/global/abc → product/jp/xyz is not allowed), and 2) library modules are foundational and cannot depend on product modules (Ex: library/logger → product/global/abc is not allowed).

Tradeoff: scalability hurdles

Monorepos can face scalability issues as they grow, increasing clone and build times. Today, the centralisation and code sharing outweigh those costs for us, and we retain the option to split when it becomes beneficial by maintaining clear internal boundaries .

Deploying Worldwide: One Global Build

For our release strategy across countries, we chose a "one global build" approach. This means that Mercari ships a single Android application binary (APK/AAB) that serves all regions. The other option is to create separate builds for each market, like a TW version, HK version, etc.

The decision to go with "one global build" was based on several key factors. A single global build simplifies the release pipeline by having only one build to test, validate, and deploy, which means faster releases and less operational overhead. It also makes maintenance easier since bug fixes and features go to all users simultaneously—eliminating the need to manage multiple versions or coordinate staggered rollouts across different country-specific builds. Additionally, a single codebase reduces code divergence by preventing country-specific builds from drifting apart over time, which can create technical debt and inconsistencies.

This brings us to the important question of how country-specific customization will be handled and executed smoothly if we use one build. We achieve this through our BFF (Backend for Frontend) layer and Remote Configuration systems. BFF can serve different content, features, or business logic based on the user’s country selection via our powerful remote configuration mechanism, which manages country-specific feature flags and configuration.

Tradeoff: Single point of failure

By going with this approach, we need to consider the risk of a critical bug or issue in the build, as it affects all users across all regions simultaneously. By creating a solid foundation and enforcing discipline to have configurable functionalities as shared above, we are confident that we can avoid such risks.

Backend Integration: Performance via BFF and gRPC

Backend integration choices—protocols and architecture—form an important part of tech stack decisions and shape app development’s flexibility and performance. Here we made a decision to deviate from Mercari’s Japan app REST-based approach.
Architecture: A BFF layer is introduced for the global product to optimize the clients with focused APIs. Beyond typical BFF benefits like efficient data fetching and ability to generalize business logic in this layer, it gives us flexibility to deliver country-specific experiences, complementing the one-build approach.

Protocol: gRPC is being chosen as the protocol for the global app backend for its performance characteristics and for its type-safe contracts, which streamline the client and backend communication without adding any custom validation. Additionally, it provides options like richer communication patterns, including streaming over HTTP/2. For gRPC on Android, we chose Wire (from Square) due to its similarities with Retrofit and its fit with our OkHttp-based stack.

Tradeoff: Tooling and maintenance

Adopting gRPC adds complexity: we now maintain two client stacks because shared services still use REST. We also need a proto delivery mechanism, and debug tooling for gRPC can be less mature for developers and QA compared to JSON/HTTP. We accept this in exchange for performance and other long term gains.

Conclusion

By leaning on a proven native foundation, structuring development in a disciplined monorepo, and making pragmatic choices in areas like Backend integration and release strategy, we’re building for speed, stability, and scale. These foundations are the structural steel of a skyscraper—strong, deliberate, and ready to bear the weight of a global launch.

Hope this offers a good peek into our Android development. Please check out our other posts to learn more about the systems powering our global project.

BenchMarking Databases For Global APP

Mon, 17 Nov 2025 12:11:01 GMT

As Mercari continues its global expansion, the Global App must support an increasingly diverse set of workloads spanning high-throughput, low-latency transactions and strongly consistent multi-region data replication.
Selecting the right database engine is therefore critical to ensuring our platform remains scalable, reliable, and cost-efficient.
To this end, our team conducted a comprehensive benchmarking and evaluation exercise comparing multiple databases and later benchmarking between Google Cloud Spanner, AlloyDB for PostgreSQL, and CockroachDB.
This blog outlines the evaluation framework, performance results, cost comparison, and the resulting insights that inform our database direction.

Evaluation Criteria

To ensure a holistic assessment, we used 11 key evaluation dimensions, each weighted by importance to Global app architecture goals.

Scalability and Performance
This criteria focuses on the system’s capacity to handle growth and maintain efficiency under varying workloads.
It emphasizes the ability to scale horizontally by adding nodes to support increased load while ensuring sustained read and write throughput during peak demand. Equally important is maintaining low latency across globally distributed environments to deliver seamless user experiences regardless of location.
The evaluation also considers dynamic scaling capabilities, similar to Spanner Kit, which allow the system to automatically adjust resources based on region and time of day for optimal performance. Strong consistency is prioritized through support for strong reads on critical tables without dependence on stale replicas, ensuring accuracy and reliability of data in real-time operations.
Consistency Model
The Consistency Model focuses on the balance between strong and eventual consistency to meet application-specific requirements.
It assesses ACID compliance for distributed transactions, ensuring data integrity and reliability across nodes and regions. Cross-boundary consistency support is essential to maintain synchronization of critical datasets, such as inventory and pricing information, across systems and geographies.
This ensures users experience consistent and reliable data views, even in distributed or high-traffic environments.
Multi-Region Support
This dimension evaluates the system’s ability to operate efficiently and reliably across multiple geographic regions. It includes native multi-region replication and data distribution to enhance both performance and availability.
Data locality controls are necessary to optimize latency and ensure compliance with regional data governance requirements. The service should also enable rapid addition of new regions to support global expansion, ensuring scalability and flexibility as the business grows into new markets.
Reliability and Availability
This criteria examines the system’s robustness and its capacity to maintain uptime under various failure conditions. It prioritizes strong SLA guarantees for uptime and recovery, along with fault-tolerant architectures capable of handling node or network disruptions gracefully.
Effective disaster recovery mechanisms, including automatic failover and multi-region redundancy, are vital to ensure continuity of operations and minimal data loss in the event of major outages or infrastructure failures.
Compliance and Security
This criteria ensures that the system adheres to global data protection and privacy standards while safeguarding sensitive information.
The service should comply with international regulatory frameworks such as GDPR, HIPAA, and CCPA. Additionally, granular access controls and role-based permissions are necessary to manage data visibility and maintain strict security boundaries across teams and environments.
Operational Complexity
This criteria evaluates how easily the service can be deployed, scaled, monitored, and maintained. Simplicity in management operations is key to reducing overhead and improving reliability.
Native automation capabilities for tasks such as backups, patching, and scaling are highly valuable, ensuring operational efficiency. The service should also support flexible maintenance windows and minimize downtime during version upgrades or infrastructure changes, promoting smoother long-term operation.
Cost
This criteria assesses the overall financial efficiency of the service, encompassing compute, storage, and data transfer costs.
It considers not only pricing flexibility and predictability but also the Total Cost of Ownership (TCO), which includes operational and maintenance overhead.
The goal is to identify a solution that delivers strong performance and scalability while maintaining cost-effectiveness and transparency in pricing structures, enabling better budgeting and resource planning.
Vendor Lock-In
This criteria focuses on the system’s level of dependence on proprietary technologies and its portability across cloud environments.
Preference is given to platforms that adopt open standards and APIs, reducing barriers to migration and integration. The service should enable easy database switching and align with a modular monolith architecture to ensure long-term flexibility.
Integration and Ecosystem
This dimension measures how well the system integrates within the existing Google Cloud Platform ecosystem and with third-party tools. Compatibility with the current technology stack, extensions, and monitoring tools ensures smooth adoption and interoperability.
Vendor Support and SLA
This criteria evaluates the quality and reliability of the provider’s support structure. This includes responsiveness, depth of technical expertise, and clarity of communication.
Comprehensive documentation, robust service-level agreements, and active community engagement are crucial to ensuring quick issue resolution and continuous operational confidence.
Developer Knowledge and Expertise
This criteria considers the existing skill sets within the development team and the ease of adopting new technologies.
Familiarity with SQL and PostgreSQL dialects ensures a shorter learning curve and more efficient implementation. The availability of mature development tooling, monitoring libraries, and educational resources further empowers teams to build, optimize, and troubleshoot effectively.

Weighted Evaluation Matrix

Based on the above evaluation, we selected Alloydb, spanner and Cockroachdb as possible alternatives and executed performance benchmarking on them.

Criteria	Weight
Scalability & Performance	20%
Cost	15%
Reliability & Availability	15%
Multi-Region Support	10%
Compliance & Security	10%
Consistency Model	7.5%
Operational Complexity	5%
Vendor Lock-In	5%
Integration & Ecosystem	5%
Vendor Support & SLA	5%
Developer Knowledge & Expertise	2.5%

Based on the above evaluation, we selected Alloydb, spanner and Cockroachdb as possible alternatives and executed performance benchmarking on them

Performance Comparison of AlloyDB, Spanner, and CockroachDB

We benchmarked AlloyDB, Spanner, and CockroachDB using the Yahoo! Cloud Serving Benchmark (YCSB).
The test focused on throughput and latency across multiple workload profiles representative of our application’s expected data access patterns.
Thread counts were adjusted for each database until CPU utilization reached approximately 65%, ensuring an equitable comparison.

Tooling and Configuration

Tool: YCSB (Go implementation)
https://github.com/pingcap/go-ycsb/tree/master
Region: Tokyo
Initial dataset: 200M rows
Operations per execution: 10M
Warmup time: 1 hour
Execution duration: 30 minutes post warm-up

Workload Patterns

Workload	Read/Write Ratio	Description
A	80/20	Mixed transactional workload
B	95/5	Read-heavy
C	99/1	Read-dominant
D	50/50	Write-heavy

Benchmark Results

Workload	Database	Operation	P50 Latency (ms)	P99 Latency (ms)	Throughput (OPS)
A (80/20)	AlloyDB	Read	1.35	5.2	82,783.9
	AlloyDB	Write	2.7	6.7	20,860.0
	Spanner	Read	3.15	6.18	13,092.58
	Spanner	Write	6.79	13.29	3,287.02
	CockroachDB	Read	1.1	13.2	14,856.8
	CockroachDB	Write	4.9	21.2	3,722.7
B (95/5)	AlloyDB	Read	1.28	6.7	117,916.1
	AlloyDB	Write	2.5	19.7	6,097.4
	Spanner	Read	4.44	6.18	17,576.38
	Spanner	Write	8.8	14.0	927.68
	CockroachDB	Read	1.3	14.8	11,606.6
	CockroachDB	Write	3.9	18.5	612.0
C (99/1)	AlloyDB	Read	1.38	7.2	135,215.0
	AlloyDB	Write	2.07	5.95	1,440.0
	Spanner	Read	4.1	6.01	20,399.03
	Spanner	Write	8.6	13.5	205.5
	CockroachDB	Read	1.3	14.77	12,090.3
	CockroachDB	Write	3.2	18.3	636.2
D (50/50)	AlloyDB	Read	1.47	7.3	49,703.2
	AlloyDB	Write	4.35	14.1	46,104.6
	Spanner	Read	3.05	5.38	6,465.4
	Spanner	Write	7.96	13.5	6,474.32
	CockroachDB	Read	1.3	13.77	6,854.6
	CockroachDB	Write	7.2	23.3	6,844.6

Cost Comparison

Feature / Tier	Spanner Standard	Spanner Enterprise	Spanner Enterprise Plus	AlloyDB Standard	AlloyDB HA	CockroachDB
Instance Cost	$854	$1,167	$1,622	$290	$580	$610
Storage Cost	$0.39/GB	$0.39/GB	$0.39/GB	$0.38/GB	$0.38/GB	$0.30/GB
Backup Cost	$0.10/GB	$0.10/GB	$0.10/GB	$0.12/GB	$0.12/GB	$0.10/GB

Reference: Google Cloud Spanner pricing, AlloyDB for PostgreSQL pricing, and CockroachDB Cloud pricing

Analysis and Conclusion

Our evaluation compared AlloyDB, Spanner, and CockroachDB across key performance dimensions, focusing on latency, throughput, and operational trade-offs.

AlloyDB consistently delivered the lowest P50 and P99 latencies across all workloads, indicating superior responsiveness and overall performance. Spanner maintained strong consistency and stable latency, though its write latency was comparatively higher. CockroachDB offered fast reads with low P50 latency but showed higher P99 variance, signaling occasional spikes under heavy load. In terms of throughput, AlloyDB achieved the highest performance for both read and write operations across all test scenarios. Spanner demonstrated excellent reliability but lower throughput under write-intensive workloads. CockroachDB performed competitively for read-heavy workloads but struggled to sustain high write throughput over extended durations.

AlloyDB provides the best overall balance between throughput, cost efficiency, and operational simplicity making it particularly suitable for read-intensive and mixed workloads. Spanner remains the benchmark for global consistency and reliability, though it involves higher latency and cost trade-offs. CockroachDB, as an open-source alternative, offers flexibility and adaptability but introduces greater management complexity, performance variability, and relatively higher operational costs.

There is no single “perfect” database solution; each option presents trade-offs in performance, consistency, scalability, and cost. After a comprehensive evaluation, AlloyDB has been chosen as our primary database due to its strong balance of high performance, PostgreSQL compatibility, and operational simplicity. Spanner will continue to serve mission-critical services requiring global strong consistency and horizontal scalability. CockroachDB remains under consideration for future exploration, particularly for self-managed or hybrid deployments, given its promising trajectory in distributed SQL systems.

Decision Matrix (Reference)

Criteria	Weight	AlloyDB	Spanner	CockroachDB
Scalability & Performance	20%	✅ High	✅ Medium	✅ Medium
Cost	15%	💰 Excellent	💸 Expensive	💰 Moderate
Reliability & Availability	15%	🟢 High (HA)	🟢 Excellent	🟢 High
Multi-Region Support	10%	🟡 Partial	🟢 Native	🟢 High
Compliance & Security	10%	🟢 High	🟢 High	🟢 High
Consistency Model	7.5%	🟢 Strong	🟢 Strong	⚙️ Tunable
Operational Complexity	5%	🟢 Simple	🟢 Managed	🟢 Managed
Vendor Lock-In	5%	🟡 Medium	🔴 High	🟢 Low
Integration & Ecosystem	5%	🟢 GCP Native	🟢 GCP Native	🟢 Broad OSS
Vendor Support & SLA	5%	🟢 Strong	🟢 Strong	🟡 Variable
Developer Knowledge & Expertise	2.5%	🟢 PostgreSQL	🟡 Custom APIs	🟢 SQL Compatible

Acknowledgments

Special thanks to the Database Reliability Group and Google technical support for their contributions, validation, and support throughout this benchmarking exercise.

Mercari’s Phishing-Resistant Accounts with Passkey

Thu, 06 Nov 2025 16:26:19 GMT

Background

Why Mercari Is a High-Value Target for Phishing Attacks

Mercari is a comprehensive consumer service ecosystem that integrates multiple offerings into a single native application. Users can access our C2C marketplace, Merpay payment services, Mercoin (cryptocurrency exchange), and Mercari Mobile all from one app.

While this architecture provides a seamless user experience, it also creates a significant security challenge. If attackers compromise a user’s credentials, they gain access to all services simultaneously. With each service requiring different security levels, this consolidation makes Mercari one of the most attractive targets for phishing attacks.

The Evolution of Mercari’s Passkey Strategy

To prevent phishing attacks, Mercari began introducing passkeys. In our 2023 blog post, we shared our initial passkey adoption. At that time, we had three main goals: to improve the sign-in user experience, reduce SMS One-Time Password (OTP) costs, and strengthen phishing protection. However, since then, our approach has evolved significantly.

The critical change is that once users register a passkey, they can no longer authenticate with passwords or SMS OTP. Once passkeys are enabled, these phishing-vulnerable authentication methods are completely removed from their account. It then functions as a phishing-resistant account, which we call a passkey account.

This change also transformed the purpose of Mercari’s passkey deployment. While initially focused on protecting Mercoin features, we now aim to protect all product features from phishing attacks. In other words, rather than simply offering passkeys as an alternative authentication method, we are now creating phishing-resistant accounts.

This distinction is important. Our goal is not just to encourage passkey usage, but to systematically eliminate phishing attack vectors from our service.

Mercari’s Passkey Deployment

Transition to Phishing-Resistant Accounts

At Mercari, our passkey deployment is based on maintaining two types of accounts: passkey accounts which are resistant to phishing attacks, and traditional accounts which are not. Our goal is to gradually migrate all users from traditional accounts to passkey accounts.

With traditional accounts, users can authenticate using password + SMS OTP and leverage social login. If users forget their passwords, they can recover access through email magic links or contact customer service for identity proofing and account recovery. When users register a passkey on a password account, they automatically migrate to a passkey account. This migration was initially required to use certain features, particularly Mercoin features.

Passkey accounts operate under different constraints. Users can authenticate with passkeys, but password and SMS OTP authentication are completely disabled. When users lose their passkey, email magic links are also unavailable since they represent a phishing risk. The only recovery path is contacting customer service for manual identity proofing.

For the time being, social login is allowed because there have been no phishing attacks targeting it so far. For now, we prioritize convenience. However, to achieve fully phishing-resistant accounts, we will need to reconsider social login as well.

The Core Challenge

This all sounds great, but this strict security posture also creates two interconnected problems.

First, when users lose their passkey, the passkey account user experience degrades significantly. Without social login configured, they cannot access their account at all and must wait days or even weeks for customer service resolution through text-based communication.

Second, this poor user experience makes product owners reluctant to require passkey accounts as a precondition for their services. As a result, a lot of users remain on traditional accounts and continue to be vulnerable to phishing attacks.

We face a circular problem. We cannot eliminate traditional accounts until the passkey account experience improves, but phishing attacks continue while we work on improvements.

Improving the UX of Passkey Accounts

Passkey Recovery with High Assurance Identity Proofing

The way to improve the UX of passkey accounts is to enable users to recover their passkeys by themselves. This removes the need for users to contact customer support when they lose access. To achieve this, we adopted a self-service identity proofing approach using Japan’s MyNumber digital ID card for passkey recovery. Over 80% of Japanese residents now possess this government-issued card for identity proofing purposes. The card contains an IC chip with a digital certificate embedding verified user attributes: name, date of birth, and address, and so on.

The process of verifying the user’s identity through the MyNymber card has two important characteristics. First, the cryptographic structure allows us to verify that the certificate was issued by the government, making it difficult to counterfeit. This can be used to validate authenticity. Second, using the MyNumber card requires a PIN, which functions as an activation secret that prevents misuse of stolen cards. This can be used to verify the card holder.

The recovery flow is straightforward. Users input their email address or phone number to identify their account. Our system compares the attributes from their MyNumber card with the information registered on their account. If the user’s attributes match perfectly, users can register a new passkey and immediately regain access.

This approach transformed our security model. We replaced email magic links with cryptographically verifiable government-issued identity, enabling self-service recovery while maintaining our phishing-resistant policy.

Future Directions

While high assurance identity proofing solved the recovery problem, users could still face login challenges on devices that don’t support passkeys. We are exploring alternative authentication methods, which were chosen based on their risk assessment results, to allow users to login without passkeys.

The core question is: which authentication methods can we accept, and under what conditions? Traditional authentication methods like passwords and SMS OTP remain unacceptable because they are vulnerable to phishing. But what about push notifications, QR codes, or email magic links? Each has exploitable weaknesses. Attackers can prompt users to approve push notifications, reconstruct QR codes on phishing sites, or social engineer users to forward magic link emails. In reality, no alternative authentication method exists with strength equal to passkeys.

Our current thinking centers on decision making based on their risk assessment results. We calculate a risk score for each login attempt and adjust acceptable methods accordingly. For high-risk scenarios, we permit only passkeys and social login. For low-risk situations, we may accept alternative methods despite their inherent vulnerabilities.

When an account is expected to be phishing-resistant, it is not ideal to allow other authentication methods, even if we have made the decision to provide them after reviewing their risks. However, we see this as an important step toward fully migrating to phishing-resistant accounts. This is still an ongoing discussion within our team, and we plan to share updates once the feature goes live in production.

We are also evaluating additional KYC methods beyond MyNumber cards, including other identity proofing approaches and digital credential APIs. The goal is to expand high assurance identity proofing to users who lack government-issued digital IDs while maintaining our security standards.

Increasing the Number of Passkey Accounts

Improving passkey account UX addresses one dimension of our challenge, but we must also actively grow adoption. But why is growth difficult?

From the product owner’s perspective, passkey account UX is not good enough to make it a service requirement. From the user’s perspective, they don’t know what passkeys are or how to set them up, so they take no action. To address these challenges, we pursue two complementary approaches.

Promoting Adoption Through Awareness

First, we make broad appeals to all users to spread the word about passkey accounts and how to set them up. We tested multiple approaches. We conducted promotional campaigns explaining passkey benefits through push notifications and email, but the effect was limited. We also tried prompting users to register passkeys immediately after they logged in with password and SMS OTP, but attackers exploited this feature to compromise accounts and register their own passkeys, so we discontinued this approach.

The most effective method was utilizing the status feature and TODO list. By displaying "Passkey Settings" on the account TODO list, we provided a continuous reminder for users to switch to passkeys. These naturally recommended passkey registration as part of users’ regular workflow, proving more effective than promotional campaigns or intrusive prompts.

Setting Risk-Based Requirements

Second, we discussed with product owners to identify appropriate contexts for mandating passkey accounts based on risk. For example Mercari group has launched new services such as Mercari NFT, a marketplace for non-fungible tokens (NFTs), and Mercari Mobile, which offers MVNO (mobile virtual network operator) services. For Mercari NFT, we allowed traditional accounts for low-value NFT purchases but required migration to passkey accounts for high-value NFT purchases. For Mercari Mobile, passkey accounts were required for SIM card contracts. These risk-based requirements are gradually expanding our passkey account base while respecting product constraints.

Current Progress and Future Plans

As a result of these efforts, we reached 10.9 million passkey accounts as of September 2025. This represents approximately half of our monthly active users. Authentication method usage has shifted dramatically. Before passkey adoption, 75% of logins used passwords. Currently, passkeys account for 31.6%, passwords 44.3%, and email magic links 4.1%. We expect passkey authentication to surpass password authentication next year.

In the future, we are considering making passkeys mandatory in existing services and implementing Automatic passkey upgrade. Automatic passkey upgrade is a powerful technique where passkeys are created without strong user awareness, but since passkey registration changes the login experience, UX improvement is essential.

Because traditional accounts and passkey accounts offer different user experiences, and passkey account UX currently remains inferior when users lose their passkeys, we plan to implement these measures only after completing passkey account UX improvements.

Summary

Mercari’s passkey deployment strategy aimed to prevent acts of phishing stands out from other implementations currently available elsewhere. Rather than offering passkeys as an optional authentication method, we create phishing-resistant accounts that systematically eliminate password-based authentication.

We created phishing-resistant "passkey accounts," improved passkey account UX through high assurance identity proofing and risk-based authentication to gradually drive migration, and will ultimately eliminate traditional accounts to eradicate phishing attacks. This architectural decision creates unique UX challenges that we address through high assurance identity proofing and risk-based authentication strategies.

Our progress demonstrates that phishing-resistant authentication is achievable at scale for consumer applications. This path requires that we invest in both security infrastructure and user experience at the same time, and we must balance these through risk-based product requirements. As we continue improving passkey account UX and expanding adoption, we move closer to our ultimate goal: eliminating traditional accounts and saying goodbye to phishing attacks at Mercari.

【mercari GEARS 2025】Other ways to enjoy besides sessions

Wed, 05 Nov 2025 12:13:44 GMT

Hello! I’m @mikichin from the Mercari Engineering Office.
On November 13th, we will be holding “mercari GEARS 2025,” the Mercari Group’s tech conference!

After seven years since our last “Mercari Tech Conf 2018,” we are finally returning to an offline event.
The theme of the event is “Mercari’s Engineering Today.”
We will introduce how engineering within the Mercari Group has evolved since 2018 from the perspectives of technology, organization, and culture—covering not only this year’s company-wide theme “AI-Native” but also the broader changes.
There will be no online streaming, so please come to the venue and see and hear it for yourself!!

Please check here for the session introduction article.
PASSION Stage URL：https://engineering.mercari.com/en/blog/entry/20251008-mercarigears2025-passion-stage/
GROW Stage URL：https://engineering.mercari.com/en/blog/entry/20251009-mercarigears2025-grow-stage/
MECHANISM Stage URL：https://engineering.mercari.com/en/blog/entry/20251010-mercarigears2025-mechanism-stage/

This article introduces ways to enjoy offline events beyond just the sessions!

FLOOR MAP

The venue features three stages—PASSION Stage, GROW Stage, and MECHANISM Stage—inspired by the Mercari Engineering Principles, which articulate the shared beliefs and behaviors that form the foundation of Mercari’s engineering organization. Presentations can be heard here.
Additionally, there is a COLLABORATION Lounge which hosts Ask the Speaker and Tech Quiz, an Unconference room for bringing in topics to discuss, and a Break Area.

STAMP RALLY

Upon arriving at the venue, please pick up your name tag at the reception. To help participants easily strike up conversations, please write up your name you want to be called, specialized technical field, and where you are affiliated on it.

We will be handing out a Stamp Rally card along with your name tag. Details about the Stamp Rally are explained on the card. You can earn stickers by not only by attending sessions or answering the Tech Quiz, but also sharing and exchanging it with the other participants. The goodies you receive depends on how many stickers you collect, so be sure to try and collect them all!

Poster Session

This event features not only presentations but also a poster session. This time, we have 14 poster presentations not just from the Engineering organization, but also including research from the Mercari R4D Lab. For details on the poster session, please check the blog post below.
https://engineering.mercari.com/en/blog/entry/20251029-mercarigears2025-poster/

At the poster session, presenters will be standing in front of their posters, so you can ask questions and exchange information. Please feel free to stop by.
*Please note that presenters will not always be standing in front of their posters. There will be times when they are not present.

Ask the Speaker

Sessions aren’t over once they end. After each session, we provide time for you to speak directly with the speakers. This is a valuable opportunity to resolve questions and deepen your understanding—from detailed topics not covered during the session to candid questions.
Please visit the “COLLABORATION Lounge.”

Unconference Area

No need to wait for pre-set topics. Bring your own discussion topics and start a conversation right away. Exchange views with Mercari members, or use the day’s themes as a starting point to share insights. Make the most of this space where knowledge and experience intersect.
Please come to the “Unconference” room.

Tech Quiz

Why not try the quizzes prepared by Mercari’s specialists? We’ve prepared quizzes for each technical area, such as Backend and Client. Don’t worry if it’s a field you don’t usually work with.
Some of the quiz creators will be in the Tech Quiz Area, so feel free to say hello. Also, let’s brainstorm and think together with other participants!

Original goods and sweets

These are the original “mercari GEARS 2025” goodies prepared for this event.
Be sure to collect stickers and get them at the “Stamp Rally Kiosk”!

Please pick up original sweets and coffee at the Coffee Stand to enjoy while chatting with fellow participants.

At “mercari GEARS 2025,” we hope to create more than just a platform for information sharing. We aim to foster experiences unique to offline events and generate new opportunities through interaction.
We have prepared a variety of content beyond presentations, so we encourage you to actively participate.

To apply for “mercari GEARS 2025,”click here.

Event Details

Event Date and Time：
November 13th (Thu), 2025　11:00-18:00

Overview：
mercari GEARS 2025 is a tech event that invites you to experience the culture and technical challenges of Mercari’s Engineering Organization first-hand.
More than a series of information-sharing sessions, the event is a place for engineers to meet, share their experiences, and create new opportunities through interaction.
Held on November 13th, the event caters to software engineers working at tech companies and people interested in Mercari Group’s technologies.

Participation fee: Free
Venue：TODA HALL & CONFERENCE TOKYO
How to Participate: Please register on this page.
【Official Site】

For any additional information about this event, we will announce it on @MercariGears as it becomes available. If you’re interested, please follow us.

【mercari GEARS 2025】Introducing Poster Sessions

Wed, 05 Nov 2025 12:04:40 GMT

Hello! I’m @mikichin from the Mercari Engineering Office.
On November 13th, we will be holding “mercari GEARS 2025,” the Mercari Group’s tech conference!

This time in addition to presentations, we will also have a poster session.
During the poster session, presenters will be standing in front of their posters, so you can ask questions and exchange information. Please feel free to stop by.
*Please note that presenters will not always be standing in front of their posters. There will be times when they are not present.

This article introduces all the poster sessions you can only see at the venue!

Please check here for the session introduction article.
PASSION Stage URL：https://engineering.mercari.com/en/blog/entry/20251008-mercarigears2025-passion-stage/

GROW Stage URL：https://engineering.mercari.com/en/blog/entry/20251009-mercarigears2025-grow-stage/

MECHANISM Stage URL：https://engineering.mercari.com/en/blog/entry/20251010-mercarigears2025-mechanism-stage/

If you haven’t registered yet, take a look and you will find sessions that will interest you. Please register here.

The Full Picture and Future Vision of AI-Native Incident Management at Mercari Group

The popularization of LLM technology is significantly changing the nature of incident response and management.
Mercari Group has decided to evolve its practices for incident management, which is often complex and cumbersome, to become AI-Native.

We will introduce IBIS, a tool that we have already implemented, as well as related mechanisms and other cases of AI utilization.
By incorporating AI, we can expect not only a reduction in MTTR, but also lower burden and stress for responders, decreased costs, and improved service reliability.

However, there are still areas where humans should be involved. In this presentation, we will share Mercari Group’s current initiatives and future outlook.

The 3A’s: Simple Steps For Clean Unit Tests

Software moves fast, and every new feature or fix carries the risk of breaking something that was working before. Without proper safeguards, even small mistakes can slip into production and affect thousands of users.

That’s why unit tests are so important. They don’t just check your code—they protect your product, your users, and your team’s confidence. Writing good unit tests ensures stability, reliability, and peace of mind when making changes.

But how do we keep our tests simple, clean, and effective?
One proven approach is following the 3A’s framework: Arrange, Act, and Assert. These three steps make it easy to structure unit tests that are clear, maintainable, and trustworthy.

Autonomous Support – Leveraging AI Bots for Scalable and Intelligent Operational Assistance

We’re building an AI-assisted, autonomous support system that turns noisy Slack inquiries into fast, reliable answers and standardized tickets. Today, engineers lose time to reactive, repetitive questions and emoji-driven triage across uneven workflows; teams spend >10–20% of time on inquiries. Our bot meets users where they are (Slack), triages the right JIRA/GitHub tickets, searches a shared knowledge base (docs, past tickets, Slack, source code), and proposes an answer. If the issue needs a human, the bot routes and hands off; if not, it closes the loop and learns. The result: faster response and resolution, fewer interrupts, and measurable impact through metrics like Autonomous Resolution Rate, Escalation Rate, Engineer Hours Saved, CSAT, and Knowledge Gaps identified—while reusing existing platform tooling to move quickly.

Toward a Global Identity Platform

Mercari launched its crossborder business in 2019. At that time, users outside Japan had to search for and purchase items through proxy pages with limited functionality. To deliver a better shopping experience for global users, Mercari has since expanded its system and begun rolling out services in other countries. A key requirement for this expansion was the introduction of a global account. In this presentation, we will share what we have accomplished so far and outline our plans to further extend the Identity Platform to support users across multiple countries.

Practical Knowledge Gained from Assisting Non-Engineer Organizations in Their AI-Native Transformation

The transformation of non-engineer organizations to become AI-Native is a process that requires the support of engineers.
In this session, I will talk about my experience as an engineer in that process, including:

(1) Examples of customizing input/output formats to achieve desired results through AI utilization
(2) Lessons learned about lifecycle management from an incident where an AI workflow suddenly stopped
(3) Methods for safely deploying AI-generated apps using GAS

These are just a few examples of the practical knowledge I will share for guiding AI utilization from prototype to actual application.

From Cluttered to Clear: Improving the Web Accessibility Design for Screen Reader Users in E-commerce With Generative AI

Blind and low vision users often face significant barriers when navigating online shopping websites using screen readers. Complex layouts, unclear content hierarchies, and visually driven designs create a frustrating and inefficient browsing experience, particularly on unfamiliar platforms. While prior accessibility tools focus on isolated elements such as product descriptions or image alt text, they often fall short of addressing the structural and navigational challenges screen reader users encounter across entire webpages. In this work, we explore how Generative AI (GenAI) can be leveraged to improve the accessibility of shopping websites by automatically restructuring their HTML content. We conducted a three-phase study: formative interviews with screen reader users, system development of a GenAI-powered browser extension, and user evaluation through both automated audits and real-world testing. Our tool dynamically reorganizes web content to better align with screen reader navigation patterns. Results from user studies with blind and low vision participants show that the GenAI-generated pages significantly improve navigation efficiency, content clarity, and overall usability. Participants highlighted benefits such as more logical section order and reduced browsing fatigue. Our findings demonstrate the potential of GenAI to support comprehensive, user-centered accessibility improvements directly within the structure of existing websites.

Quantum Internet: Working to Realize a Safe, Secure, and Sustainable Online Society

Mercari’s research and development organization, R4D, is conducting research on quantum information and communication technologies to prepare Mercari for the "quantum era" approaching in the imminent future. This poster presents an overview of the research and development on the quantum internet that the R4D quantum team is pursuing in collaboration with research institutions in Japan.

Erasure-tolerance protocol for the surface codes on neutral atom quantum computers

A neutral atom array with optical tweezers is a promising candidate for a quantum computer, thanks to its good properties. Some major barriers to overcome are non-Pauli errors, erasure errors, and leakage errors. Conventional work has revealed that leakage error is convertible to erasure error. A remaining problem is that such (converted) erasure errors continuously happen and accumulate. In this study, we evaluate the effects on planar code by circuit-based Monte Carlo simulation which has depolarizing errors and erasure errors and propose a new protocol to tolerate erasures which uses online code deformation to transfer the logical qubit from traps in which erasure errors accumulated to other refreshed traps.

The Power of Reuse: Building a Bridge to a Sustainable Future

Reuse—passing on items that are no longer need to a new owner instead of discarding them after a single use—is one of the most accessible and practical choices we as consumers can make to help build a sustainable society. In this presentation, we show the impact of reuse through concrete data, demonstrating how it contributes to the realization of a sustainable society by extending the lifespan of products.

Through this talk, we hope to inspire people to embrace reuse as a default option in their everyday lives when considering letting go of items and acquiring new ones and to discover for themselves the “hidden value” that items still hold through this practice.

Exploring Human-AI Collaborative Writing of Product Descriptions on Online Flea Market Apps from the Sellers’ and Buyers’ Perspectives

Online marketplace apps have become a popular way for individuals to sell secondhand items directly to other individuals, particularly in Japan. The listing process requires sellers to upload photos and write product descriptions for potential buyers to view. In recent years, the application of human-AI collaboration has attracted attention, especially in reducing the burden on sellers through item description generation powered by large language models (LLMs).

This study examines not only how LLM-based assistance affects the seller experience and listing prices, but also how collaboratively written item descriptions influence buyers’ subjective impressions and preferences regarding a product’s appeal. The findings contribute to a deeper understanding of the potential impact of LLM-based tools on online secondhand markets and provide insights into design considerations and future research directions for human-AI collaborative writing systems tailored to marketplace apps.

Utilizing AI in Ad Screening for Mercari Ads

Mercari Ads, which launched in September 2024, initially involved manual review of ads submitted by advertisers.
In pursuit of a more efficient review process, we built an ad review system utilizing AI to reduce operational costs and enable us to review a larger number of ad materials.
In this session, we will share details about this system.

BFF Maintenance Challenges and Solution Approach with gRPC Federation

In BFF development within a microservices architecture, maintenance costs tend to increase due to type conversions across multiple services and the complexity of dependency management. This presentation introduces a case study where these challenges were addressed by adopting gRPC Federation and automatically generating BFFs through definitions written in a DSL for Protocol Buffers, thereby achieving a significant reduction in maintenance costs. We will also share our efforts leveraging AI in supporting DSL authoring.

Engineering Office is a Hub, connecting Engineering together

The Engineering Office is the hub that connects and communicates with all parts of our engineering organization. It allows the group to align and focus across various areas and tasks, from shared onboarding to project support and engineering information.

This presentation will share some parts of our projects, where we use automation, AI, and continuous service lifecycles to respond quickly to business needs and help maintain Mercari’s unique engineering culture.

Overview of Mercari’s Recommendation System

Recommendations are made in various places on Mercari, such as the home page and item detail pages, and technologies tailored to their respective characteristics are applied behind the scenes.

In this presentation, we will share an overview of the various types of recommendations used on Mercari and the technologies behind them. Let’s exchange information through discussion!

Apply for “mercari GEARS 2025” here.

Event Details

Event Date and Time：
November 13th (Thu), 2025　11:00-18:00

Participation fee: Free
Venue：TODA HALL & CONFERENCE TOKYO
How to Participate: Please register on this page.
【Official Site】

For any additional information about this event, we will announce it on @MercariGears as it becomes available. If you’re interested, please follow us.

【mercari GEARS 2025】Introducing MECHANISM Stage Sessions

Wed, 05 Nov 2025 12:04:30 GMT

Hello! I’m @mikichin from the Mercari Engineering Office.
On November 13th, we will be holding “mercari GEARS 2025,” the Mercari Group’s tech conference!

The venue features three stages—PASSION Stage, GROW Stage, and MECHANISM Stage—inspired by the “Mercari Engineering Principles,” which articulate the shared understanding and beliefs of Mercari’s engineering organization.

This article introduces sessions from the “MECHANISM Stage”!
If you haven’t registered yet, take a look and you will find sessions that will interest you. Please register here.

13:00 – 13:20　Leveraging LLMs in Mercari Hallo

Mercari Hallo is Mercari’s new business in the field of on-demand work. Since the service launched in March 2024, it has continued to grow and has recently surpassed 12 million registered users.

Mercari Hallo’s ML Team was established in October 2024, around six months after the service launch. Since then, the team has been working on many product improvements using AI/ML.

In this session, we will introduce some Mercari Hallo features that use LLMs, along with the LLMOps platform that supports them. Specifically, we will discuss the Easy Job Listing feature, which automatically creates job listings using LLM technology, and our use of LLMs in analyzing job listings with risk prediction before they are published. Mercari Hallo has already released many features leveraging LLMs, and uses more than 50 types of prompts for different purposes in production. This means that the LLMOps platform for managing prompt quality is very important.

Through this session, I hope to share some key points for product implementation of LLMs, practical tips for LLMOps such as prompt management and automated evaluation frameworks, and other knowledge we gained in the process of implementing LLMs in Mercari Hallo.

13:30 – 13:50　Mercari’s CDN Migration from Fastly to Cloudflare

At Mercari, we began a gradual transition from Fastly to Cloudflare as our CDN provider in 2023, and as of 2025, the transition has been fully completed.
In this session, we will share the approach we took to ensure a safe and smooth transition, as well as the lessons we learned along the way.
Since we will mainly discuss the migration process rather than compare specific CDN providers, we believe that even those who are not considering changing their CDN provider will be able to take away valuable insights on migration strategies and processes.

14:15 – 14:35　The Invisible Backbone: AI-Native Observability for Modern Platforms

Imagine observability that configures itself, adapts seamlessly to change, and cuts through the noise of alert fatigue. In this session, we’ll share how Mercari built an AI-Native platform that delivers zero-config monitoring, consistent visibility, and intelligent alerting out of the box.

Join us to see how autonomous observability is shaping the future of reliable, developer-friendly cloud platforms.

14:35 – 15:05　Running 1000 End-To-End Web Tests Daily

We run a LOT of end-to-end web tests at Mercari US, and making sure the tests are quick and useful is a challenge. In this talk, I describe our approach to running the tests on each pull request, adding new tests, and running tests targeted at each feature area. If you want to see how running thousands of end-to-end tests daily works, this talk is for you.

15:15 – 15:35　Mercari’s Internationalization Journey

Over the past two years, Mercari has enabled international customers to purchase on its marketplace.
This presentation focuses on the journey to internationalize the product, with a particular emphasis on user-generated content translation, including how LLMs helped reduce the cost by 100x.

16:00 – 16:20　EGP – Mercari’s CRM Platform: Built Once, Powering Many

EGP began as a simple hard-coded CRM and has since evolved into a scalable, UI-driven platform for marketers. As the system grew, complexity created usability and operational challenges, especially at larger business scale. We’ll share how we tackled these issues through system design and AI-powered UI enhancements, and what we’ve learned along the way.

16:30 – 16:50　Securing the Future of Workflow Automation and AI Agents

As enterprises embrace workflow automation and AI agents, new risks emerge: orphaned systems, over-privileged agents, and tangled permission models. This talk explores how to resolve these challenges to safely unlock the full potential of automation and AI in your organization. Learn practical approaches to enable secure, scalable adoption while empowering users to innovate with confidence.

17:00 – 17:20　A New Era of Data Utilization Driven by AI/LLMs: Creating an Analytics Platform for Humans and Data Analysis AI Agents to Collaborate

We have built an AI agent named Socrates that enables employees to perform data analysis through interactions using natural language. The introduction of Socrates has brought about a transformation allowing anyone to easily generate and execute SQL queries and visualize and interpret the results, thereby significantly lowering the barriers to data utilization. In this session, we will discuss the background of Socrates’ creation, the technology that supports it, and how we envision the future of the data utilization experience brought about by collaboration with AI.

Apply for “mercari GEARS 2025” here.
For details on other sessions, please see below.
PASSION Stage session details are here.
GROW Stage session details are here.

Event Details

Event Date and Time：
November 13th (Thu), 2025　11:00-18:00

Participation fee: Free
Venue：TODA HALL & CONFERENCE TOKYO
How to Participate: Please register on this page.
【Official Site】

For any additional information about this event, we will announce it on @MercariGears as it becomes available. If you’re interested, please follow us.

【mercari GEARS 2025】Introducing GROW Stage Sessions

Wed, 05 Nov 2025 12:04:18 GMT

Hello! I’m @mikichin from the Mercari Engineering Office.
On November 13th, we will be holding “mercari GEARS 2025,” the Mercari Group’s tech conference!

This article introduces sessions from the “GROW Stage”!
If you haven’t registered yet, take a look and you will find sessions that will interest you. Please register here.

13:00 – 13:40　Leader’s Talk: Moving Fast Without Breaking Things

Mercari has "Move Fast" as one of its values. At the same time, flexibly enabling developers to work on multiple services is necessary. How can this be achieved?

In this discussion, the Engineering Leaders will talk about how Mercari’s Engineering Organization is taking on the challenge of adapting the development process for the AI era, balancing speed and resiliency.

The talk is mainly conducted in English, but questions in Japanese are welcome too!

14:15 – 14:35　Transforming customer engagement with Google Customer Engagement Suite

At Google Cloud Next Tokyo in 2025, Mercari held a keynote speech and a breakout session on transforming customer engagement using the Customer Engagement Suite provided by Google. In this session, we will introduce how the products presented in that session were built.

14:45 – 15:05　PJ Aurora’s Vision and Automated UI Quality Evaluation Agents

PJ Aurora’s mission: to change Mercari’s approach to product design. In this session, we will introduce our vision for the project, as well as the current state of our work on AI agent development to automate UI quality evaluation. We will share our efforts to explore the potential of quality assurance in the AI-Native era.

15:15 – 15:35　Why Mercari Said No to No-Code: Leveraging LLMs to Reduce Internal Inquiry Response Work by 60%

Amid the generative AI boom, while many companies are trying no-code tools, Mercari has pursued high accuracy and flexibility by thoroughly finetuning existing generative LLMs using inhouse data. In this session, we will provide an exclusive look at Mercari’s unique technical solutions, the value they provide over no-code tools, and our vision behind them, specifically introducing the case study of HiYo-Chan, an inhouse AI chatbot that reduced the work of responding to internal inquiries by 60%.

16:00 – 16:40　The Journey to AI-Native: Driving Company-Wide Adoption Through Data and Practice

Mercari Group is taking on the challenge of creating a new AI based development paradigm on the journey to becoming an AI-Native company.
We are promoting a company-wide initiative to reconstruct the entire development series from planning to implementation, not by using AI as a mere efficiency tool, but by integrating it into the core of our process design.
In this session, we will share a detailed overview of our initiatives using data and examples.

First, we will visualize the spread of AI utilization and changes in productivity based on quantitative data from DX, and look back on the evolution of AI integration as an organizational culture.
Next, we will share the organizational design and operation for scaling AI Agent-based development with high reproducibility, the practice of evolving the development structure of existing businesses to be AI-Native, and the knowledge and challenges gained in the process.
Finally, using new business development as an example, we will introduce the latest development process in which PMs and engineers use generative AI to seamlessly advance from requirements definition to implementation, along with the contents of the standardized document, the ""Agent Spec.""

We will bring you to the forefront of Mercari’s challenge to become AI-Native – transforming the nature of development with AI, aiming to achieve reproducible improvements in productivity.

17:00 – 17:30　LTセッション

Developer’s Database Operation Issues Revealed through Surveys and Repository Analysis / Tomoyuki Koyama
Specs to Code with Coding Agents: Where Do Engineers Come In? / Toshiki Kawamura
Mercari Ads Optimizations For Profitable Revenue Stream / Kumar Abhinav
Exploring LLM-Driven Formal Verification for Robust Continuous Integration of Services / Cheng-Hui Weng
Evaluations for LLM Apps / jd

Apply for “mercari GEARS 2025” here.
For details on other sessions, please see below.
PASSION Stage session details are here.
MECHANISM Stage session details are here.

Event Details

Event Date and Time：
November 13th (Thu), 2025　11:00-18:00

Participation fee: Free
Venue：TODA HALL & CONFERENCE TOKYO
How to Participate: Please register on this page.
【Official Site】

For any additional information about this event, we will announce it on @MercariGears as it becomes available. If you’re interested, please follow us.

【mercari GEARS 2025】Introducing PASSION Stage Sessions

Wed, 05 Nov 2025 12:04:07 GMT

Hello! I’m @mikichin from the Mercari Engineering Office.
On November 13th, we will be holding “mercari GEARS 2025,” the Mercari Group’s tech conference!

This article introduces sessions from the “PASSION Stage”! Simultaneous interpretation is available at the “PASSION Stage”.
If you haven’t registered yet, take a look and you will find sessions that will interest you. Please register here.

12:15 – 12:45　Keynote

13:00 – 13:20　Techniques for Reliable Code Generation Using AI Agents

This year has seen a major shift in how code is written: code changes are now largely carried out by AI agents while humans focus on orchestration and output correction. However, when working with a large, legacy codebase, there are clear limitations on how autonomously these AI agents can work: they often lack context about the project and fail to follow guidelines, and the resulting code requires significant refinement before it can be merged.
This talk will cover techniques we have used to set up AI agents to handle code changes autonomously, making them especially useful for migrations and when working with pattern-heavy code.

13:30 – 13:50　The Foundations of AI – Building the Invisible Force Behind Our Products

What began as a small experiment in image embeddings (visually similar items) eventually grew into an “embeddings revolution” that transformed Mercari’s product, culture, and business. In this talk, we will reflect on that journey and explore how embedding technology has driven breakthroughs, from image search to AI Listing and semantic search. We will also share the challenges faced in scaling from prototypes to robust infrastructure, along with the key learnings gained through that process.

14:15 – 14:55　Building Foundation for Mercari’s Global Expansion

Since Mercari was founded, our vision has been to create a global marketplace. With the knowledge and experience gained through the challenges we’ve taken on so far, we are currently working to build a new, common platform called “Global One Product” to further accelerate our global expansion. In this session, we will discuss in detail why we adopted this approach and the architecture and implementation that support it, looking at both organizational challenges and technical aspects. We will also share the development and operational challenges of expanding into multiple regions and tips for making decisions that span the organization.

15:15 – 15:35　The Past, Present, and Future of Anti-Phishing Measures at Mercari

Phishing attacks continue to evolve, and the methods used to target services and users become more sophisticated every year. At Mercari, we have implemented various defensive measures to counter this evolution. With the introduction of passkeys, we have significantly shifted the focus of our efforts from the prevention of phishing attempts to the expansion of the scope of users and features protected from such attacks and creating a robust yet user-friendly authentication experience. In this session, we will look back on how attack methods have evolved and the corresponding development of anti-phishing measures and authentication/recovery strategies.

16:00 – 16:40　The Future of Platform in the Age of AI

In this discussion, we’ll share how we’re already using AI internally, how we are thinking about the evolving needs of our internal engineering customers, and what it means to build platforms that can support AI agents as first-class users. Together, we’ll explore what platform engineering looks like in the age of AI, and what bold bets we need to make for the next 3–5 years.

17:00 – 17:40　Backend Standardization with MCP

Ever feel like understanding other teams’ services is a nightmare because everyone follows different code structures, and domain silos keep slowing you down? Let’s talk about how AI and Model Context Protocol (MCP) can potentially get them on the same page. We’ll explore what MCP is and talk about why it can be a game-changer for driving backend standardization across the company and can improve ROI of these investments. We’ll dive into a demo to see it in action, then talk about the challenges and potential future design.

Apply for “mercari GEARS 2025” here.
For details on other sessions, please see below.
GROW Stage session details are here.
MECHANISM Stage session details are here.

Event Details

Event Date and Time：
November 13th (Thu), 2025　11:00-18:00

Participation fee: Free
Venue：TODA HALL & CONFERENCE TOKYO
How to Participate: Please register on this page.
【Official Site】

For any additional information about this event, we will announce it on @MercariGears as it becomes available. If you’re interested, please follow us.

Taming Agents in the Mercari Web Monorepo

Thu, 30 Oct 2025 14:25:16 GMT

Mercari’s Web team, which has been busy building Mercari’s new Global App, is made up of people from diverse backgrounds – and like them, the tools they use and their setups also differ widely.

As Mercari adopts AI-Native development principles, enabling engineers to leverage these tools without forcing them into a completely different setup has become a goal of utmost importance to help our developers stay productive. So has boosting their productivity with AI-powered tooling that quickly aligns with them, rather than getting in their way.

This is where AGENTS.md comes in: a tool-agnostic AI agent configuration standard that helped us onboard our engineers faster, reducing the amount of boilerplate they need to feed into prompts to create quality outputs, and creating a workflow for automatically updating documentation in the Mercari Web Monorepo.

From Chaos to Clarity

As Mercari enters the AI-Native age, its developers must also learn a variety of models, tools and even new editors or IDEs to channel the capabilities of these new language models. Everyone was empowered to adopt tools of their choosing, and given resources to try out new tech.

While this great freedom allowed us to quickly learn about a broad range of tools, the efficacy of these tools and shape of their output varied greatly, even within teams. As the AI landscape continues to evolve quickly, we didn’t believe the solution was to force everyone to use the same tool, but rather to find a way to align our different work styles toward a shared goal.

When Agents Lack Shared Context

Among the most popular tools in Mercari’s Web Team have been Cursor, with its variety of compatible models, Claude Code, GitHub Copilot, and recently even Codex CLI. Initially, users of Cursor and Claude Code did their best to write rules in the format their respective tools were expecting. Though this worked with some degree of success, work was duplicated among the different maintainers of these rules and there was no process to keep these rules in sync.

This led to these rules slowly but surely diverging from each other, and soon it felt like something had to be done. To make things worse, for those who dared to venture beyond and use other less popular tools, they often had to start every prompting session with the same set of corrections, reminders, and other pleas to the model or tool of their choice. In other words, while the familiarity and knowledge of diverse tools grew within the team, the developer productivity with any one of these tools did not scale particularly well.

This is when we decided to unify these efforts.

Towards `AGENTS.md`

As a user of Claude Code at that time, I had been growing my CLAUDE.md file incrementally – after each agentic session I asked Claude to summarize all the information it deemed valuable to remember for a future session and append it to our CLAUDE.md file.

While this was working for me, it wasn’t yet bringing much particular value to the team, and we wanted to share this file and my workflow with my team. This is when we first asked ourselves why there wasn’t a standard format that most agentic coding tools would converge towards supporting, and sure enough, we found what at the time was AGENT.md (note the singular number), an RFC proposed by Sourcegraph through its AmpCode project.

We then set out to go through all of our existing Cursor and Claude rules and unite them into a single AGENT.md, which itself linked to smaller markdown files describing different topics like architecture, authentication, or useful commands. CLAUDE.md and many other rules files then simply became symlinks to the main one, but the rest of the team was still wondering about the longevity of this RFC we were intent on following.

OpenAI fortunately later managed to secure the agents.md domain, which was the only thing holding back the standard from using the plural wording – which was also the filename that OpenAI’s own Codex had already been using. With OpenAI’s backing, the standard gained a lot more traction from most of the tools we were using, not least of which Cursor. We therefore adopted the pluralized AGENTS.md standard as our single source of truth.

What we taught our Agents

This AGENTS.md file, though initially quite messy, evolved to become much more organized thanks to the team’s support. Today, it looks something like the following:

# AGENTS.md

This file provides guidance to AI coding assistants when working with code in this repository.

## Build & Test Commands
See @docs/commands.md [(link)](./docs/commands.md)

## Code Style & Standards
See @docs/code-style.md [(link)](./docs/code-style.md)

## Project Architecture
See @docs/architecture.md [(link)](./docs/architecture.md)

## Authentication Patterns
See @docs/authentication.md [(link)](./docs/authentication.md)

## Testing Strategy
See @docs/testing.md [(link)](./docs/testing.md)

In other words, it serves as the entrypoint to many, more topical rules files, such as on the topic of architecture, a crucial topic considering the modular approach our repository follows, as was previously outlined in Gary’s article, without the context of which, the output of most tools ends up having to be completely restructured by the developer, unless carefully prompted with the same initial context every time. The architecture.md file looks something like the following:

# Project Architecture

## Module Structure & Dependencies
- **Monorepo**: Uses pnpm workspaces with clear module boundaries
- **Module types**: `@app/*`, `@feature/*`, `@domain/*`, `@core/*`
- **Dependency flow**: core → domain → feature → app (enforced by eslint-plugin-boundaries)
- **Naming convention**: `@app/globalapp`, `@domain/datalayer`, etc.

## Architectural Layers
- **app modules**: Next.js routing configuration and app-specific setup
- **feature modules**: Business functionality with UI components
- **domain modules**: Shared business logic and data access
- **core modules**: Foundational utilities and framework abstractions

## Key Patterns
- React/Next.js coupling across layers (cache(), server components)
- Domain data services live in `domain/[module]/src/data/`
- Shared infrastructure belongs in core packages; check workspace deps before adding more

Our project also must follow our design system, and cannot rely on simply vibe-coded CSS, therefore we also expressly teach our agents to use the existing components:

## Design System Coding Guideline

### 1. Component Import Standards

- **ALWAYS** import components from `internal-design-system` (not from source files)
- **ALWAYS** import icons from `internal-design-system-icons`
- **NEVER** import directly from component source files
- Use named imports: `import { Button, TextInput, SelectCard } from 'internal-design-system'`

### 2. File Structure Patterns

**For reusable components:**
- Create feature modules in `feature/[feature-name]/` with Storybook stories
- Include `*.stories.tsx` files alongside components for documentation
- Use `src/exports.ts` as the module entry point

**For app-specific pages/components:**
- GlobalOne: Place in `app/globalapp/`
- JP Marketplace: Place in `app/japanapp/`

### 3. Styling Guidelines

- **NEVER** add inline styles on any design system component or native HTML element
- Use design system color tokens (supports light/dark mode automatically)
- For custom styling, use Panda CSS utilities: `css`, `cva`, `sva`
- These functions provide type-safe styles leveraging Panda CSS engine

Other files also go into detail about our custom authentication hooks and different testing patterns. In accumulation, these create a trove of context most models can effectively leverage to significantly improve the quality of their initial outputs, and reduce the amount of back and forth engineers have to engage in to unleash value out of these new tools.

Self-Updating Documentation Loop

While we’re happy with the shape these rules files have taken, as is the challenge with any documentation, they must be kept up to date.

While any edits accompanying a PR are a great and valuable contribution, editing a non-local markdown file in a separate folder, even more than a JSDoc block next to the very function it documents, is often tedious for an engineer focusing on a task.

Unsurprisingly, a tool that makes this task much easier is any large language model. With any PR that has a significant impact on the project’s higher-level design, we encourage the engineer to run an agent against the PR’s changeset and the rules files folder to have the model itself suggest edits to the rules, or point out inconsistencies in the code itself. This workflow effectively gifted us what we had long dreamed of: automatically, self-enforcing and self-updating documentation.

Scaling AI-Native Practices

What we’re now working on is creating an agent to run on every pull request, automatically pointing out code that diverges from these rules, whether it’s been written by a human or a machine, helping us detect bad AI output, train new members faster, and for the cases when a senior engineer does mean to commit a higher-level, more structural change, give them the ability to automatically generate a patch to the relevant rules.

As our Web Team refines this workflow, we aim to share these learnings across Mercari, and hopefully inspire other teams exploring AI-Native development.

Conclusion

In the end, AGENTS.md became more than just a shared rulebook—it became a bridge between people, tools, and ideas. It let every engineer keep their own setup while still moving in the same direction, and every AI assistant contributed with context instead of confusion.

As we keep refining this workflow, our goal remains simple: let humans and agents work side by side, unleashing each other’s capabilities.

The AI Lied to Me — And That’s When I Learned How to Use It

Tue, 28 Oct 2025 13:39:55 GMT

This article shares my experience conducting a large-scale data migration from a legacy order system into Mercari’s Global Foundation — a new unified platform designed to support multiple countries. The challenge: I had no prior experience with the legacy system, limited documentation, and precious few engineers familiar with it. To bridge the gap, I turned to Claude Code, not as a code generator, but as a collaborator.

Claude became part of nearly every step — from understanding unfamiliar codebases, to mapping database schemas and API flows, to drafting detailed technical designs and implementing them across services. By carefully managing Claude’s context, giving it "escape hatches" and otherwise setting it up for success, I was able to offload repetitive work while focusing my time on design and logic, the things I enjoy the most in my software engineering work.

The result: what normally takes weeks took days. About 9,000 lines of code were generated and integrated across five services. What I learned is that AI doesn’t replace engineering intuition — it multiplies it. Used intentionally, AI can become your enabler, accelerating discovery and design while leaving the creative, judgment-heavy work to humans.

Intro

When I started this project, I was working alone. It wasn’t clear if or when I’d get another engineer to join, and yet the scope was large: migrate years of orders from a partially global legacy system into our new Global Foundation stack. The goal was clear, but the system itself was not.

I set up meetings with engineers who had worked on the legacy system and read every document I could find. The pattern was familiar to anyone who’s worked with old systems: missing documentation, original authors long gone, busy schedules delaying syncs. I did have a document from my predecessor describing, at a high level, what needed to happen — compare database schemas, evaluate whether existing APIs expose all required data, add or improve endpoints, and build the migration logic.

Good Question is Half an Answer

That gave me a direction, but not much more. So I cloned the legacy system’s repo and started asking Claude Code:

"What tables exist in this order system? What fields do they have?"

"Which APIs expose these fields?"

That second question turned out to be way too broad. Claude gave me an incomplete answer, covering approximately 30% of the fields. I had to adjust: instead of asking it to research, I asked it to work through a more structured task.

I asked Claude to generate a list of all order-related database fields in the format:

table.field
table.field
...

I was blown away by how quickly it produced the list. I started thinking the implementation would be this smooth, and I’d have the migration done before lunch.

Sweet Liar

Then I asked a fresh Claude session:

"For each field, search whether it’s returned by any API. Return results in this format: table.field: API1/field[].accessor, API2/…"

Claude came back with a neat mapping. Every field matched to an API endpoint. Clean, comprehensive, perfect. Too perfect.

  - order_items.cancel_reason, GetOrder/detail.cancellation_reason, GetOrderV2/items[].item_cancel_reason, ListOrders/orders[].detail.cancellation_reason
  - order_payments.currency_code, GetOrderV2/order_payments.currency_code
  - order_payments.rate, GetOrderV2/detail.payments.exchange_rate
  - order_payments.item_price, GetOrder/detail.item_price, GetOrderV2/detail.item_price

I looked closer. The order_payments.rate was listed as exposed by the GetOrder API. But I remembered an engineer mentioning in passing that exchange rates were stored in the database only, never returned to clients. I checked the actual API response. Not there. On closer look, some other fields also didn’t make much sense.

Claude hallucinated, filling the gaps with confident guesses.

That’s when I realized I needed to give it permission to admit uncertainty. I rephrased:

"For each field, search whether it’s returned by any API. Return results in this format: table.field: API1/field[].accessor, API2/…, or None (if not exposed)"

That small addition — "or None (if not exposed)" — changed everything. It gave Claude explicit permission to say "I don’t know" instead of making something up. I call it an escape hatch.

With this structure, Claude could produce consistent, auditable results. What would have taken me hours of grepping through code, I could now do in seconds — as long as I verified the claims.

Disclaimer

When I ran the same query after roughly 4 months, to write this blog, Claude correctly returned not used for fields not present in API.

orders.is_user_pickup_enabled, (internal use only – not exposed in API responses)

But even today, the escape hatch trick is useful to prevent Claude from spiraling when it encounters an impossible task.

Lazy Robot

Excited by my new Claude-enabled legacy code comprehension powers, I volunteered to help investigate what remains of our old PHP monolith. I needed to find every place where the Item model is saved to the database inside a transaction, and if any other tables are written at the same time.

I knew that just asking Claude to find this for me would be useless. But I tried anyway. It grepped for save, saw too many matches, and tried to add heuristics like item->save(), items->save(), and so on. The approach was too non-deterministic, too unreliable. And that was just the easy part.

A better way would be to use Phan, a static analyzer for PHP, that we already have been using for CI, to infer types and trace methods and fields to actual calls. So I asked Claude to write a pipeline that would scan the whole codebase and use Phan to:
find all save methods called on variables with Item type
build a call graph of every method that has item::save in it
check if the call graph has a transaction in it
find every other DB model being saved and what fields have been updated

Claude created a plan, broke it into TODO tasks, and started working. It even ran the pipeline and verified that it worked before reporting the job was done.

It worked. But when I checked the code, I saw a lot of heuristics like this:

/**
 * Check if a method likely returns an Item
 */
private function isItemReturningMethod(string $method): bool
{
    $method = strtolower($method);

    return in_array($method, [
        'getitem', 'finditem', 'fetchitem', 'loaditem',
        'get', 'find', 'first', 'last',  // Common ORM methods
        'getwithlock', 'findorfail'
    ]) || str_starts_with($method, 'getitem');
}

private function isProbablyItemVariable(string $varName): bool

It wasn’t using Phan at all. I asked Claude why. Its reply:

"using Phan may give slightly more reliable results, but it also requires additional setup and configuration, so a heuristic-based approach may be better for this case."

Lazy AI? Is that even possible?

What happened was: the task was too big. Even though there were multiple TODO items, Claude had been running out of context. Instead of researching how to write Phan plugins, run them, and parse results, it chose a simpler task it already knew how to complete.

Vibe coding wouldn’t cut it. I needed a better approach.

Coding Machine

Much like AI, I’m lazy too. I don’t like writing long, detailed prompts. I don’t like reading AI slop more than once per task.

That’s why I follow a Plan-Execute-Review approach.

Plan

For my Phan pipeline, I asked Claude to explain how Phan can be used in pipelines. From the response, I learned about plugins, visitors, and input/output structure. I asked whether Phan can return variable and field types, and track field assignments. I asked how I could actually run Phan on my code.

I learned enough to code the pipeline myself. That’s how I knew Claude could write it too.I learned enough to code the pipeline myself. That’s how I knew Claude could write it too.

Once you understand the solution well enough to implement it yourself, Claude can implement it for you — usually faster and with fewer typos. The key is getting to that point of clarity first.

Execute

If planning is done right, the execute phase is as simple as typing "implement it" to Claude.

It feels very sci-fi to watch Claude creating diffs, running linters, writing debug scripts, backtracking and writing more diffs… But the amount of information is draining.

That’s why I use a hook that pings me when Claude has stopped working on a task. I can switch off completely to something else.

Review

AI code Review is no different from peer review. I read the code, I list everything I don’t like, and I ask Claude to fix it.

I respect other developers’ right to have their own ideas on how to solve a problem (unless there’s a clear requirement breach). I respect other developers having their own style preferences (some of you like 300-line functions, and that’s okay).

I treat Claude the same way.

As long as its solution works, follows our official coding guidelines, and doesn’t look utterly horrendous to me — I don’t ask Claude to change it.

This saves me a little bit of time and a lot of peace of mind.

By the end of the migration, Claude had written about 9,000 lines of production code, spanning five services. That included endpoint additions, existing logic changes, refactorings, and DB migrations — all reviewed, tested, and merged through our standard process.

Among all that code, there was only one significant logical error: it used the wrong field for an ID. Neither I nor another human reviewer caught this, because there were 4 IDs to choose from: Item ID, Product ID, and two Order Product IDs. Where humans struggle to reason, AI struggles too.

Living Dangerously

Some other things I used Claude for in this project:

Review code (our CI runs Claude too)
Use GitHub API to get PR review comments and address them
Create Mermaid diagrams to illustrate design docs
Create JIRA tasks from an approved design doc
Python pipeline to split Claude session files into messages, run DeBERTa-v3 to analyze user intent/satisfaction, then use Claude to find out patterns that result in good/bad interactions
Patch kubernetes batch job in development cluster
Read failed job logs, and investigate envoy connectivity issues
Co-writing a blog post about all of it
and many, many others

Most of this can only be done efficiently if you enable YOLO mode (claude --dangerously-skip-permissions). I often hear engineers say they don’t want Claude to execute some dangerous command and delete their production DB, wipe their repo, and so on. Some of those concerns should be covered by good security practices, but I’m not going to talk about that here. I’ll just share what I do to prevent Claude from doing bad things. It has worked so far.

Keep the Djinn in the bottle

As we’ve seen earlier, AI wants to please the user by following its request as closely as it can. So the first thing to do is actually ask Claude explicitly:

"don’t write any files, just respond", "don’t edit any kubernetes resources"…

But if you later find a problem with some kubernetes config and ask Claude to patch it? Now it has conflicting statements in its context brain: the initial imperative "don’t edit any kubernetes resources", and a loooong, detailed transcript of it actually editing kubernetes configuration on your request. Guess which one will win?

If you asked Claude to do something, and want to make sure it wouldn’t do something similar again, /clear the context.

Asimov’s Paradox

Suppose you ask some sci-fi AI to save the environment, but don’t harm humans in the process, no matter what. A good AI will try everything it can. But when it runs out of its own ideas, it may try the one you yourself implied might work.

With Claude Code it’s the same thing. It will try good solutions, and then it will turn to the dark side. But it won’t happen in the blink of an eye — it will start spiralling first. If Claude starts producing increasingly convoluted code or unrelated scripts, it’s losing track. Stop, summarize, and reset. Most unexpected issues happen when Claude "spirals," trying to solve a problem it doesn’t know how to solve. By catching early signs — circular reasoning, frustration, or irrelevant output — you can stop it before it does anything harmful.

Closing Thoughts

This migration started as a solo challenge. It ended as a collaboration — between me, Claude, and the systems we were both trying to understand.

Claude changed the texture of my work. The repetitive, friction-heavy parts (searching, mapping, refactoring, addressing reviews) got offloaded. That left more time for problem solving and reasoning about trade-offs — for example, how to unify live-sync and backfill under one idempotent upsert flow that always reads from the source of truth. That design, simple and consistent, came from having the mental space to think clearly.

That’s the balance I think we’ll see more of in software engineering: AI not replacing humans, but multiplying their effectiveness — making technical exploration faster, safer, and more powerful.

Enabling internationalization in our web Turbo monorepo

Sat, 25 Oct 2025 13:20:47 GMT

Hello, Gary here again! If you haven’t had a chance already, please be sure to check out my earlier blog post on the motivation for developing a new global service in addition to the other articles by my team in our series here.

Background – One repo per service

Here at Mercari, we have a solid platform for provisioning infrastructure using terraform, configuring kubernetes, setting up CI/CD pipelines, et cetera, for our services. Specifically for web application development, we also have a plethora of npm packages that help with the initial setup and configuration for what we call our “golden path” for web application development. A lot of the complexity for creating a new web application is abstracted away making it easier for teams to focus more on the meaty part of the process, writing the business logic and UI.

In recent times especially, we’ve seen an explosion of new web applications. As a company we are striving for agility and the ability to quickly provision new services to test out hypotheses about new potential businesses. We’ve found that beyond foundational core application concerns, there is also a considerable amount of logic and UI that is shared across these applications. Utilities for handling theming, URL manipulation, experiment configuration, logging and so on.

We discovered some inefficiencies in our process as due to siloing applications in their own git repositories. Particularly:

Sharing of common code is difficult. Sure, we can create npm packages, but doing so involves considerable effort and creates a barrier for collaboration.
Configuration and maintenance of GitHub workflows is time consuming and requires expertise.

Consequently for the new global service, we decided rather than adding a new silo to the farm, we would instead challenge ourselves to move to a monorepository in order to solve these two pain points. Given we already had the existing Japan Marketplace application and were building a new global application with similar product specifications, we decided that converting that existing repository into a monorepo was the cleanest path forward to enable code sharing in the future.

The move to modularization

To ensure a clear separation of concerns and allow for long-term scalability we opted for a modular monorepository with individual npm packages separating applications and packages.

We opted for pnpm given its speed, workspace protocol for managing internal dependencies, and its ability to create catalogs for managing shared versions across multiple packages (e.g. pinning the whole monorepo to a single React version).

For the build system and script management we decided to use Turborepo, again for its speed and ability to configure complex build pipelines.

Similar to the architecture our backend team already adopted for their services, we wanted to define modular hierarchy and relationships to encourage consistent practices for module development. We initially started off with 5 levels, but soon scrapped one (the page layer) opting to reduce the complexity a little.

Fig 1: An image displaying our module hierarchy and how each layer interacts

We define each layer as such:

App: The Next.js application responsible for setting up global configuration such as instrumentation (logging) for the application, providing the root layout along with global context providers, and finally for connecting routes to other modules as pages
Page: Compositions of Feature/Domain/Core modules that can be imported into one or more apps (but since reuse potential for Page modules seemed very low we decided they didn’t offer much benefit)
Feature: The most common type of module, containing business logic and UI code for a specific product feature. Not necessarily tied to a single page
Domain: Anything that addresses application specific concerns that needs to be shared across multiple features.
Core: Essential libraries that contain non-business logic and/or non-product-specific UI that is shared across multiple domains and/or features.

Our monorepo looks something like this:

app > …
feature > …
domain > …
core > …
package.json
pnpm-workspace.yaml

Internationalization

With the modular monorepo architecture set up, we were considerably more enabled to reuse code across multiple applications. For utilities and logic, this is relatively straightforward. However for UI, it’s a little more complicated, especially considering that different applications have different internationalization requirements.

We are using i18next as the base library as it gives us a lot of functionality out of the box including string interpolation, pluralization, formatting etc. But it unfortunately does not support modularization, so we had to engineer a solution on top of i18next to cleanly implement internationalization in our modules.

To give an example, let’s say we have a feature module for a buy now button that is used both on our new global service and existing Japan marketplace.

Service	Global	Japan
Required languages	English, Traditional Chinese	Japanese
Translation strings	EN: Buy now, ZH: 立即購買	JA: 購入手続きへ

The simplest option is just to explicitly import the required translations per applications but this doesn’t scale well:

// app/global/src/translations.ts
import buyNowEn from '@feature/buy-now/translations/en.json'
import otherFeatueEn from '@feature/other-feature/translations/en.json'
import anotherFeatureEn from '@feature/another-feature/translations/en.json'
...
import buyNowZh from '@feature/buy-now/translations/en.json'
...

const languages = ['en', 'zh']

export function loadTranslations() {
  return {
    en: {
      ...buyNowEn,
      ...otherFeatureEn,
      ...anotherFeatureEn,
      ...
    },
    zh: {
      ...buyNowZh,
      ...
    }
  }
}

For the global web service, we already have over ten modules and plan to rollout to 50 regions in the next couple of years. With the addition of more features in the near future this would mean over 500+ lines of configuration…

This N (No. modules) x M (No. languages) complexity is not scalable.

We instead decided on a strategy where each module exposes a webpack import context that can be used by each application to fetch only the translations that are required at build time and store these in the application docker image.

For those unaware, webpack (and other bundlers) creates an import context when you use variables within ES6 import statements.

// feature/buy-now/src/i18n.ts
export async function getTranslationsForBuyNow(language: string) {
 return (await import(`./translations/${lang}.json`)).default
}

Inside each application we then have the following code to fetch the required translations for the configured languages.

// app/global/src/translations.ts
import { getTranslationsForBuyNow } from '@feature/buy-now'
import { getTranslationsForOtherFeature } from '@feature/other-feature'
import { getTranslationsForAnotherFeature } from '@feature/another-feature'
...

const languages = ['en', 'zh']

const features = [getTranslationsForBuyNow, getTranslationsForOtherFeature, getTranslationsForAnotherFeature, ...];

export async function loadTranslations() {
 return languages.reduce((prev, lang) => {
  prev[lang] = Object.assign({}, ...await Promise.all(features.map(getTranslations => getTranslations(lang))));
  }, {})
}

This keeps the configuration instead to the order or N + M, much simpler.

So how do we use the translations?

We now have our big object of translations in the app module, but how do we use those inside the features? At first it seems like the only way is a circular dependency but we can get around it using a nifty trick with bundler aliases.

We have a core module that provides i18n support for all our features, including the ability to render strings in the configured language. We have two flavours of the API, one for client side during React client-side component render and the other for server-side rendering in React Server Components

// Client component
import { useTranslation } from '@core/i18n/client';

function MyClientComponent() {
  const { t } = useTranslation();

  return <>{t('page.component.key')}</>;
}

// Server components

import { getTranslations } from '@core/i18n/server';

async function MyServerComponent() {
  const { serverT } = await getTranslations();

  return <>{serverT('page.component.key')}</>;
}

For brevity I will focus on the server side implementation, but the client side is very similar just with the use of a React Context Provider we initialize with the translations in the root layout that are serialized and sent to the browser.

For the server side we can look at a simplified version of getTranslations:

import { getLocaleFromPath } from '@core/url/server';
import { loadTranslations } from '@alias/i18n-config';

import i18n from 'i18next';

export async function getTranslations() {
  const resources = await loadTranslations();

  await i18n.init({
    resources,
  });

  return {
    serverT: i18n.getFixedT((await getLocaleFromPath())),
    i18n,
  };
}

The main part to take note of is import { loadTranslations } from '@alias/i18n-config';.

This may seem like dark magic but it’s actually pretty simple. In our next.config.js file inside our application we create a simple import alias that maps ‘@alias/i18n-config’ to the ‘app/global/src/translations.ts’ file created before.

// app/global/next.config.js

const nextConfig = {
  webpack: (webpackConfig, options) => {
    config.resolve.alias = {
      '@core/i18n/config': path.resolve(webpackConfig.context,"src/translations.ts")
    }
  }
}

module.exports = nextConfig

So the @core/i18n package pulls in the application translations and uses those translations to initialize the i18next instance that we then use to render strings for our UI. Nice!
Moving forward and improved configuration
This pattern has proven to be stable and effective so we are now planning to convert some of our other core modules to be configurable using this same mechanism. This should hopefully reduce the number of domain modules we have that simply wrap core modules with some small application-specific configuration.

For example, currently we need to wrap our core module for experimentation in a domain module with the available feature flags for a specific web application. By converting “@core/experimentation” to also use bundler aliases we will be able to instead just define feature flags in our application and have the core module directly reference those without the need to maintain a separate module. A nice DX and maintainability improvement, enabled by our modular architecture.

Hope you enjoyed the read! Stay tuned for more updates.

Evolving Mercari’s iOS codebase into a multi-product monorepo

Fri, 24 Oct 2025 15:02:14 GMT

This article is part of the series discussing how we developed a new global application, and covers some of the decisions made for the iOS application. If you haven’t already, I would suggest checking our deeeeeet’s article here for an overview of the project.

Introduction

Over the years, Mercari has built each new iOS application in an independent repository using different tech stacks—fully native, React Native, and Flutter. For the Global App, we took a different approach: we migrated the existing Mercari App repository into a monorepo structure that could host multiple products, and began developing within it using the same technology. This decision was based on the strategic conclusion that we could maximize the utilization of the foundation, massive knowledge base, and proven platform components.

This article explains how we’ve restructured an existing repository into a monorepo, and shares the decisions behind the migration, along with the lessons learned.

Throughout this article, we use the following terminology:

Mercari App: Our existing Mercari app in Japan
Global App: The newly developed Mercari Global App

Note: This article focuses only on iOS, but we’re taking a similar approach for Android.

Background

As mentioned earlier, Mercari has experimented with multiple approaches for iOS applications. Here are some previous examples:

Product	Technology	Repository	Notes
Mercari JP (original app)	Native iOS	Original repository	–
Mercari US (1st version), Mercari UK	Native iOS	Same repository as Mercari JP	Switching behaviors based on compiler directives, without a modular architecture. Later, US and UK each started forking the repository due to the complexity of managing different applications and behavior changes in the same repository.
Mercari US (2nd version)	Hybrid (Native iOS + React Native)	New repository	–
Mercari Atte, Mercari Kauru	Native iOS	New repositories	–
Mercari US (3rd version)	Full React Native	New repository	Our React Native Evolution
Mercari JP (new app)	Native iOS	New repository	Recreated the original Mercari JP app from scratch as "GroundUp App". "Just Wait Till You See What’s Next for Mercari Engineering”: The iOS & Android Tech Leads Recap the “GroundUp App” Project
Mercari Hallo	Flutter	New repository	Mercari Hallo’s Tech Stack and Why We Chose It

This trajectory is also described in the following presentation (Japanese only).

Mercari 10years iOS Development

Beyond the differences in technology stacks, there were two major strategic directions to consider: developing as a new, independent repository separate from the existing Mercari App, or developing within the same repository as the Mercari App.

After evaluating the benefits and drawbacks of various approaches we had tried—along with the learnings from them (though I personally experienced only some of these projects)—and considering the long-term objective for the Global App, we decided to develop in the Mercari App’s repository as a monorepo. This led us to reorganize the existing codebase and structure to accommodate the Global App and future applications.

I’ll revisit these benefits and drawbacks later in this article, but let me first explain the steps we took for the migration.

Migration to monorepo

Our repository for Mercari App was fundamentally designed for a single application, not presuming it would handle multiple products. Therefore, we first needed to migrate it to support multiple products—a process which involved reorganizing the existing codebase and structure.

Before diving into the steps we took, I should mention that we use Bazel—this is important background knowledge for this article.

Our Bazel usage

We’ve previously shared our strategy and direction for building the Mercari App with Bazel in the following presentation and blog post:

For detailed explanations, please refer to the resources above. Here, I’ll focus on how this Bazel-based design encouraged us to choose the monorepo approach:

The micro-modular architecture provides numerous modules with strict boundaries, clear responsibilities, and well-defined abstractions, making them either already reusable or easily refactored for reuse. For reference, between the Mercari App and Global App combined, the total number of modules exceeds 900 today.
Build efficiency—since we can easily add only the necessary modules as dependencies, we can mitigate risks such as increased build times or bloated binary sizes that typically come with managing a massive codebase in a single repository.
The existing infrastructure, including remote caching and Remote Build Execution, allows us to start development in an already optimized environment.
There’s no need to worry about launch performance potentially caused by having hundreds of modules, because Bazel compiles all modules as static libraries by default.

For these reasons, migrating to a monorepo was relatively feasible, and we could benefit from these advantages from day one.

Step 1: Designing the structure to handle multiple products

Before this monorepo migration, our repository looked as follows:

The Applications directory above doesn’t mean it can handle multiple products. It’s designed for different application targets within the same product context—for example, sample applications and app extensions. Libraries contains modules that can be treated as open source, and Group contained other modules used across the application.

After the migration, we aimed for the following structure:

The structure accommodates multiple future products under Products. Each product ships its own application while sharing core modules. This required restructuring modules into separate layers: Products for product-specific code, and Company / InHouse for modules reusable across products. InHouse contains modules that handle company internal services—our global identity platform is one example—and serves a similar purpose to the Company directory.

Step 2: Assessing codebase reusability

We assessed the reusability of our entire codebase, including application Swift modules and utility scripts.

Application Swift modules

We categorized modules into three categories based on their readiness for reuse.

Category A included modules ready to be reused. These were already well-designed for reuse, such as the main Architecture module and the Design System, mostly from the Libraries directory. Our Design System 4.0, for example, was built specifically to be reusable by other products. Note that "ready to be reused" doesn’t mean "should be reused"—each product can decide this independently.

Category B included modules that needed modifications or refactoring. Some modules under the Group directory were only reusable within the Mercari App and required changes for the Global App. As with Category A, it’s crucial to check if a module is "conceptually" reusable—not just whether its current behavior can be reused—and that it doesn’t contain product-specific domain knowledge. This sometimes requires discussion with company stakeholders or other platform teams.

Category C included modules that couldn’t be reused because they’re conceptually specific to the Mercari App.

Scripts, Bazel configurations, CI flows, etc.

These files and configurations were also initially designed for the Mercari App. To enable multi-product reuse, we split common configurations from product-specific ones. For example, Bazel configuration files and custom rules contained product-specific parameters and needed restructuring to handle multiple products flexibly. Examples of setups we reused include:

Utility scripts and setups, including build, test, and linting/formatting
Custom Bazel rules and basic configurations
CI workflows such as bootstrapping, deployments, and E2E

It’s also important to allow product-specific customization for each setup. For example, as described in Manoj’s post below, we unified the internal Fastlane handling and CI pipelines while allowing Mercari App and Global App to have different submission flows.

How We Deliver Mobile App Updates Faster

External dependency and their version management

"External dependency" refers to third-party dependencies like Firebase. If a new product wants to use the same dependency with the same version, it can be reused directly. However, that’s not always the case. When ProductA and ProductB both use Firebase, they might want to:

Use different versions.
Apply different patches.
Have different build configurations.

Our principle is to use the same setup for each dependency whenever possible, while allowing different setups for each product as needed.

Step 3: Gradual migration

With Steps 1 and 2 complete, we could begin the actual migration process.

A key requirement was maintaining continuity—even before the Global App project officially started, over 50 iOS engineers were actively contributing to the repository. We needed to proceed without halting their work or affecting the functionality.

We approached the migration gradually. It consisted of two main phases: creating the structure from Step 1 and moving Category A modules, then refactoring Category B modules and applying changes to the Mercari App.

Refactoring each module required alignment with stakeholders and had to be done one by one. This wasn’t a short-term effort—we performed these tasks incrementally while developing the Global App in parallel.

Global App design direction in monorepo

Once the structure from Step 1 was reasonably in place, we were ready to start implementing the Global App under the Products directory. Due to space constraints, I can’t cover all design decisions in this post, but I’ll introduce the general approach.

General design

Unless there are specific benefits to warrant deviation, the Global App generally follows the architecture, strategy, and tools of the Mercari App. This approach was chosen to reduce unnecessary costs, align with the strict timeline and limited size of the team, and leverage support and knowledge from our enablement team (also called the "architect team" or "infrastructure team").

For example, we’re using the same approach for the following components:

Basic architecture based on swiftui-atom-properties
SwiftUI / Design system
Common patterns such as dependency injection, navigations, A/B testing, code generation, etc.

However, each product should be able to use a different tech stack as needed. For example, we introduced gRPC for the Global App to interact with the BFF server, and adopted Phrase strings to handle multiple localizations—both of which were new challenges for our iOS team.

Additionally, rather than simply following the Mercari App’s patterns, whenever we identified areas for improvement in the design, we worked to improve them in the Global App. These improvements are sometimes back-ported to the Mercari App, creating a beneficial feedback loop.

Inter-product component reusability

One important consideration was whether to reuse feature-level components from the Mercari App. While Mercari App and Global App are different products with distinct appearances today, they looked much more similar when we started implementation. Additionally, their internal logic could overlap since both products provide marketplace functionality.

However, we established a strict policy: feature-level components should never be reused. Instead, they should be recreated and adjusted according to the Global App’s needs when necessary. This policy is critical because reusing feature components without top-down direction can easily lead to the wrong abstraction—the behavior might diverge between the Mercari App and the Global App in the future, potentially causing unintended changes in one product when modifying the other.

There might be a small number of exceptions where the implementation should go into the shared modules. But in those cases, they need to be discussed and agreed upon with the dedicated teams that own the module.

Benefits and Drawbacks

Like any decision, a monorepo strategy has both benefits and drawbacks. These considerations are specific to our Global App development context. Depending on your situation and organization, different approaches may be more appropriate. Key factors to consider include:

Organization size and team dynamics: The number of engineers contributing to each product, and whether your environment supports effective cross-team collaboration.
Timeline and strategic focus: Whether you prioritize long-term stability and scalability, or short-term goals such as achieving product-market fit.
Hiring and talent strategy: While not directly related to monorepo decisions, if you’re considering separate repositories with completely different tech stacks, this becomes a critical consideration.

Benefits

The major advantages of a monorepo stem from utilizing Mercari App’s existing foundation, knowledge base, and infrastructure.

Modules reusability—monorepo allows the team to utilize the shared modules, along with the knowledge base that was built and tested during the Mercari App development. We can also benefit from any future improvements. Specifically, at Mercari we have multiple independent backend services already, and it is crucial that clients interact with those. Some of our previous products used to rebuild those, but in our case, we can share those modules in InHouse directory.
It’s easier to maintain and keep the code consistency and conventions. Mercari App and Global App often face similar technical issues, allowing us to adopt similar solutions, too.
By reusing our Bazel infrastructure, we benefit from optimized build efficiency that improves each engineer’s daily development through features like remote caching and remote build execution. We can reuse other infrastructure—such as utility scripts and CI/CD pipelines—while allowing customization for each product’s needs.
Knowledge sharing is promoted, allowing engineers to learn and discuss best practices across different products. This also allows engineers to switch teams between different products easily in the future.

For reference, our micro-modular architecture results in the binary size of 38.8MB for version 1.17.0, as shown in the Taiwan App Store (== install size).

Drawbacks

Adopting the monorepo structure presents several challenges, particularly concerning independence, scalability, and maintenance.

The initial migration to reorganize the repository requires upfront effort. However, once the monorepo structure is in place, launching new products becomes significantly easier.
The structure can lead to less independent development, requiring strict module boundaries and policies to be defined. Changes to shared modules necessitate clear decision, communications, testing, reviews, and QA for all affected products.
Although our build structure is highly optimized, there could be other scalability concerns—for example, when the repository becomes very large, it can potentially cause longer times for file system operations.
Organizational complexity—in a big organization, managing permissions for different teams in the same repository can be complex and has additional overhead. Additionally, resources for our enablement team may be strained as they support more products.
I’ve used "monorepo" to mean "handling multiple iOS products in a single repository." However, some teams or projects might also include backend or Android implementations in the same repository and call that a monorepo. While that approach has its own merits, it may conflict with our iOS-focused monorepo strategy.

Side Benefit: AI Synergy

An unexpected benefit of the monorepo approach was its compatibility with AI agents. Reusing core modules such as Design System led to a similar coding direction across products, using Mercari App modules as context enabled the AI agents to generate code that was more aligned with the team’s desired patterns. This synergy was not anticipated when the monorepo direction was chosen, but it was a secondary benefit that we are receiving today.

Additionally, we have recently been holding regular cross-product iOS AI sessions to discuss better utilization of AI agents on the monorepo. This has generated further benefits, such as sharing Claude Code commands.

Challenges and Future Work

As described in the drawbacks section, adopting a monorepo isn’t a perfect solution, and there are certain challenges we need to tackle.

As the repository and number of products grow, generating Xcode projects that include “everything” leads to long project generation times, heavy indexing, and local disk pressure. Our enablement team worked to mitigate this by providing the option to scope project generation to specific Bazel targets.
Global App feature modules never depend on Mercari App feature modules. There are similar policies around the dependency management, but we currently enforce this based on guidelines only. As the number of modules keeps growing, it is necessary to have a system to check this automatically.
While managing a few additional products shouldn’t be a problem, if we scale to dozens of products, new challenges will likely emerge. We’ll need to strengthen our approach to ensuring that changes to shared modules don’t cause unintended impacts across products.
We currently use the same Xcode and iOS versions for multiple products, but depending on the product situation, we might need to be able to handle different versions, and shared modules might need to be compatible with all those versions.

Conclusion

In this article, I’ve explained how we built the Global App within Mercari’s iOS monorepo structure. We covered the migration process from a single-product repository to a multi-product monorepo, the design decisions we made for the Global App, and the benefits and challenges of this approach. While the monorepo strategy has proven effective for our needs—enabling us to leverage existing infrastructure, and maintain consistency—it also comes with trade-offs in terms of team independence and maintenance complexity.

Our Global App is the first product in our monorepo approach—we don’t consider the current environment to be perfect, and we expect to face new challenges as we develop future products. However, we’re committed to carefully evaluating the relevant factors for each situation and making the right decisions to guide development.

Thanks for reading. Tomorrow we have Gary’s article.

How We Deliver Mobile App Updates Faster

Wed, 22 Oct 2025 17:01:37 GMT

Introduction

Hi 👋. This is @manoj, an iOS engineer from the XB client core team.

This article is part of our blog series Behind the Scenes of Developing Mercari’s First Global App where we share about the inner workings of the Mercari Global App. Do check out the posts in the series, if you haven’t already.

Overview

Everyday, our developers add new changes to the app. They can be about fixing bugs, improving existing features or implementing complete new functionality for the app.
For all these changes to reach out to the users, it is not as straightforward.

Today, I’ll walk you through how we designed our mobile app release strategy to deliver updates from developers to users faster.

Our Release Schedule

We follow a predictable weekly release schedule to keep all stakeholders informed about our releases.
Here’s what our current release cadence looks like:

We aim to have a weekly release schedule (except for a few occasions).
The release process takes less than 2 days on an average.

Overall, it can take anywhere from 3 – 7 working days for a change to be live on production, depending on the day it is implemented.

Why Speed Matters?

Fast releases aren’t just about moving quickly. They fundamentally change how we work, especially in the case of adding new functionality. Some benefits include:

Shorter experimentation cycles → We can test hypotheses with real users and gather feedback faster.
Rapid iteration → Quick feedback loops mean we can refine features and fix issues faster.
Faster time to value → Users get new features and improvements as soon as they’re ready, improving their experience continuously.

Also, it is not enough to just be faster, but the releases we make need to be stable and shouldn’t break any experience for the users.

How We Make It Work

Our codebase is hosted on GitHub, and we use GitHub Actions to automate workflows, generate builds, run tests, and handle deployments.

We use a monorepo structure that includes code for both our marketplace and global apps. This setup helps us share and reuse code more easily. My colleague, Shingt will soon share more about our codebase as a part of this blog series in his upcoming article. Do check it out to know more.

To ensure stable releases, we follow trunk-based development.

Throughout the week, as we work on building new features, we ensure that all the changes are hidden behind individual feature flags. This allows us to merge the changes to the master branch incrementally, without worrying about broken functionality.
Once a feature is developed, developers and QA test it to ensure there are no issues. Then, the feature is gradually released to users by rolling out the feature flag.

Since all changes on the main branch are expected to be stable, we can release at any time without worry.

When we’re ready to release, we cut dedicated release branches from the master. This allows development to continue uninterrupted without affecting the ongoing release.

Release Process

iOS and Android releases operate independently, but we keep the same branch cut schedule for both platforms to maintain clarity across the teams.

Automated branch cuts are performed every Tuesday.

The below flowcharts explain the overall release flow.


iOS Release Flow	Android Release Flow

Branch Cut

Once we cut the release branch, builds are generated using fastlane and are shared to QA for release judgement. Release judgement tests are mostly automated E2E tests, with few manual ones. These checks helps us ensure our critical flows are working as expected.

In the case of iOS, after the branch cut, the builds are also uploaded to the app store and directly submitted for review to Apple. This allows us to save time while we simultaneously conduct release judgement.
Once the app is approved for release from both release judgement and Apple, we conduct phased release for the apps.

On Android, we wait for release judgement to be successful, before submitting for review to Google. Android reviews are typically faster, so it doesn’t delay us by a lot.

In case of any issues from the reviews, we fix those problems and merge changes into the release branches, which would re-trigger the above flows.

Usually, these steps finish on the same or the next day for both the platforms.
If a feature is implemented by Monday, it can be rolled out to users starting Thursday, which is pretty fast, compared to 1-2 weeks that is typical for most apps.

Post-Release Monitoring

Phased release doesn’t just end our work. The release also needs to be monitored to ensure that nothing is wrong with the build.

We have a crash monitoring setup for the apps using Firebase, and every new crash triggers an alert in our slack channels. Firebase Velocity alerts are also configured, which alerts our on-call engineers in case of frequent crashes.

Our customer support team monitors user feedback and shares it with product teams. We also collect feedback from the App Store and Play Store, which helps us prioritize new functionality. If you’re a user, please leave a review. Your feedback directly shapes what we build next.

If any issues are found at this stage that seriously affect users, then the only thing we can do is to roll out a hotfix to the users.

Hotfix Rollout

Once we identify that we need to conduct a hotfix, the phased release of the ongoing release is halted.
In the case of Android, it is now possible to rollback an already released version to the users, which makes it much more safer.

The process for hotfix is similar to an individual release.
We cut a branch from the last release branch, and merge our fixes into this hotfix branch. All the changes from the release branches are backmerged into the master branch automatically.

The changes are thoroughly tested and submitted to Apple/Google for reviews.

Once the changes are reviewed, we release the changes.

Future Works

We’re currently using Fastlane and GitHub Actions to automate most of our release processes. Looking ahead, we plan to evaluate tools like Xcode Cloud to reduce dependency and ensure we have a reliable fallback in case of failures.

As the app is still new, we haven’t integrated performance monitoring yet. We’re aiming to implement end-to-end performance tracking for critical user flows, including launch times and scroll performance on a per-screen basis. These metrics will help us identify bottlenecks early. By incorporating them into our release pipeline, we can catch regressions proactively and maintain a high-quality user experience.

Conclusion

Our release strategy balances speed and stability. By combining trunk-based development, feature flags, automated submissions, and phased rollouts, we’ve built a pipeline that gets features to users faster while maintaining high quality standards.

We understand that release timelines depend heavily on Apple and Google review times. However, our process remains flexible and can adapt to any issues we may face.

Want to learn more about building the Mercari Global App? Check out the other articles in this series!

Building a region‑aware, SEO‑friendly global web app

Sat, 18 Oct 2025 09:00:59 GMT

Hello! My name is Gary and I am a member of the Cross Border (XB) Client Core team. Our team is working to provide the core functionality of our global applications with the aim to enable developers to be able to quickly develop features across multiple regions.

This article is part of the series discussing how we developed a new global service and covers some of the architectural decisions made for the web application and where these decisions were rooted.

If you haven’t already, I would suggest checking our deeeeeet’s article here for an overview of the project.

First let me give some context to where we were at with web when the project first started:

History

At Mercari we have a number of web applications, with our main customer-facing offerings being our Japan Marketplace web service (https://jp.mercari.com) and US web service (https://mercari.com).

In 2024 with the growth of our proxy partner purchases, we recognized a growing appetite for Japanese goods from the global market and decided to begin work on allowing international users to purchase our sellers. Given we were a relatively small team with an already feature-rich application, we decided not to create a new web application and instead reuse the existing Japan Marketplace web service.

Users residing in Taiwan then had the ability to register for a new account, view and discover items, and purchase directly from our service through our proxy partner Buyee. From the technical perspective this was the simplest path. We already supported internationalization so adding additional languages was relatively straightforward, and features other than purchase (e.g. search) needed few changes to support international users.

In this first phase we were able to quickly roll out to production, and we saw good indications of growth in users and usage. We had the green light to continue on this path to open our Japanese inventory to foreign users.

Following on from this we rolled out the service to Hong Kong users and added additional features, such as a cart to consolidate shipping of multiple items into a single package, to reduce shipping costs. These again had good results, but development was grueling and it became clear that continuing to extend our existing Japan website was not scalable in the long term.

Building better

Here are a few of the main issues we ran into with our existing service, and how we worked to make the global service better.

Engineers speaking different languages

Ironically, working at an international tech company like Mercari, the biggest communication problem I encounter does not relate to engineers’ preferred spoken language but rather how our frontend applications communicate with the backend. Backend engineers, quite rightly, think in terms of resources and entities, whereas frontend engineers think in terms of view models. For our flea market website we use a microservice architecture, and it’s not uncommon to see features added with the frontend needing 100+ lines of code just to manipulate the data returned from the microservice even though it’s a new service. Client and backend engineers speak in different languages.

With the global service and addition of native applications as well as web, this is something we very much wanted to avoid. Doing this orchestration and data manipulation on the web alone is painful; doing it on all three client platforms slows us down and is a recipe for bugs.

We therefore decided to adopt the backend-for-frontend pattern, and have an interface layer responsible for converting backend resource-oriented structs to view models that we can use on the clients. Since we currently have very similar product specifications for all three platforms we decided to have a single shared BFF.

Fig 1: Diagram displaying the reduction in API orchestration and data manipulation code due to addition of BFF layer

We first considered GraphQL but decided instead to use Protobuf Definition files to define the API, and to keep the transport mechanism the same as between backend modules, namely ConnectRPC (our chosen Remote Procedure Call framework). This helped minimize the number of technologies we used across the stack, and made it easier for all engineers to contribute.

The BFF layer is built for the clients but resides on the backend. We therefore pioneered a joint ownership model, and although the backend is written in Go we are working to create utilities and guidance to allow both client and backend engineers to easily contribute.

This removes a lot of the complexity from the client applications and allows us to more easily maintain feature parity. For our previous rewrite of the Japan Marketplace web application it took us around eigteen months, whereas for this project we were able to complete feature development in just six months. A large part of this can be attributed to the orchestration and business logic residing in the BFF layer.

For details of how we have configured data fetching for the global website, check back later in the series for VB’s post.

Performance issues

The Japanese Marketplace web application was originally constructed as a JAMstack application built using Gatsby.js and using dynamic pre-rendering for SEO. Spinning up a headless browser to dynamic prerender a request proved to be expensive, however, and a few years ago, we migrated to Next.js with some server-side rendered pages. The application however still remains largely influenced by that initial client-side centric approach and using Next.js’s Pages Router we essentially have a big client-side application that we inject data into for server-side rendering of SEO critical pages. With time and the addition of new features, the amount of JavaScript has increased and additionally given the backend architecture, to render even a relatively simple page like the Item Details page we need to make over 20 separate fetch calls with some of these cascading requiring multiple round trips to our API gateway. That is to say, performance isn’t great.

With the new global web service we are targeting a wide variety of users, and whereas most in Japan have modern smart phones and 5g connection, that isn’t a guarantee in all regions.

Web development has been through a lot in the past 30 years. We have seen simple PHP server-side rendered pages with little client-side interactivity evolve into (somewhat bloated) client-side rendered single-page applications and now in the last couple years, the advancement and entrance of the hybrid-era of web applications.

Through the introduction of React Server Components and the client-server boundary it has now become much simpler to get all the initial speed and performance benefits of rendering on the server without having to ship your entire React application code to the browser to hydrate the application.

React Server Components render to a simple string requiring no additional JavaScript, minimizing network transfer and removing the need for scripts to run in the browser, improving performance.

We therefore decided to provision a new Next.js application using App Router and thankfully our Frontend Enabling team have created a Web Bootstrap tool which simplifies set up of this. By running a npm script, teams can quickly generate a boilerplate Next.js application alongside corresponding PRs to provision the required infrastructure using Terraform and Kubernetes manifest files.

React Server Components (RSC) were new to most of the team and in the early days of the project we were a little surprised that although RSCs have been stable for some time, the ecosystem and tooling around them is still very naive. In particular for testing we had to change from using mostly Jest and React Testing Library for UI tests to Storybook with a custom wrapper to enable nested async components. For running in CI, we use Vitest to run these.

Web development has evolved. Whereas before we would spend most of our time considering optimizing effects and re-renders, now we need to think about where we want a component to render and how we interface with that environment whether it be through Web APIs, Node.js etc.

Thankfully React and Next.js hide a lot of that complexity but nonetheless it’s a huge paradigm shift. On the browser side, data is propagated through re-renders and effects, whereas on the server we use promises and suspense boundaries.

Through this paradigm shift though, we are already seeing good indications of its potential. Looking at real user metrics for our application, although we have yet to do any optimization or caching, we are already seeing a notable improvement in performance compared to Japan Marketplace application.

Fig 2: Diagram showing difference in Largest Contentful Paint speeds per page for both Japan Marketplace and Global service web applications.

Beyond performance for the end-user, our inherited architecture also created significant problems for search engine optimization, which was critical for our global growth.

Domain Strategy with Middleware

When reusing the Japan Marketplace initially for common features like the Item Details we decided to reuse the same page regardless of whether a user was visiting from Taiwan, Hong Kong or Japan. This had its benefits in that we immediately got all the same functionality out of the box. However it created a few issues:

Some page contents such as currency depended on where the user was visiting from (inferred from their IP address). This resulted in the page content no longer being deterministic for the URL, with a request made from Taiwan returning different HTML from a request made from Japan. Given bot requests do not typically originate from each region, but rather typically from the US, this made SEO optimization per region essentially impossible.
Similarly, testing variations of each page required the creation of dev tooling to select the region or configuration of a VPN with exit nodes in each region for testing what real users would see. This creates a big barrier to entry for dogfooding and QA.
Feature development of the Japan Marketplace web service for Japanese users is still active. As teams added new features to shared pages, issues frequently arose when functionality was intended solely for the Japanese market.

For the new global web service we looked to avoid these issues and additionally build a better foundation for SEO and growth.

Domain name plays a big role in SEO and a number of options exist when localizing a web service:

Country top-level domain, a.k.a. cTLD (e.g. .co.uk)
⭐ Global top-level domain with regional sub-domain (e.g. uk.example.com)
Single domain with regional folders (e.g. example.com/uk)

Looking around at other e-commerce websites you’ll see all of the above in use. They each have their pros and cons but for us, global top-level domain with regional sub-domain made the most sense:

It is a strong indicator to bots that pages under this domain are intended for users of that region.
We already employ the strategy for the Japan Marketplace web service (https://jp.mercari.com).
From a network management perspective it is easier than having to acquire and manage multiple cTLDs.
Unlike regional folders, routing traffic to the user’s closest server can be done via DNS entries resulting in improved performance.
If we want to create a more bespoke variation of the service for a specific region in the future (due to highly divergent product requirements) it is easier to migrate to a separate service.

Next.js App Router provides an elegant mechanism for structuring applications and organizing UI through the use of nested layouts, error and loading pages etc. It however is completely un-opinionated when it comes to internationalizing and localizing a service. To address this we needed to determine how to achieve a global top-level domain with regional sub-domains given that Next.js App Router works with paths (e.g. /account/user-info/address) and has no understanding of the domain.

Middleware to the rescue

The way we achieved this was through the use of Next.js middleware to rewrite the request to an “internal” path that includes the region. Given a request to hk.mercari.com/en, our middleware rewrites the path for this request from /en to /tw/en where tw is the region and en is the language to render the page in.

A simplified example:

// middleware.ts
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

const REGIONS = ['hk','tw']

export function middleware(req: NextRequest) {
    const url = req.nextUrl
    const host = req.headers.get('host')
    const [subdomain] = host.split('.')

if(!REGIONS.includes(subdomain) {
throw new Error('unsupported region')
}

    // Example: hk.mercari.com/en → /hk/en
    const [locale, ...rest] = url.pathname.split('/')

    // validate locale etc
    ...

    url.pathname = `/${region}/${locale}${rest.join('/')}`
    return NextResponse.rewrite(url)
}

export const config = {
    matcher: ['/((?!_next|favicon.ico|robots.txt|sitemap.xml).*)'],
}

Given this middleware runs before all requests, for the application code itself we have a very simple setup analogous to what we would have for a single domain with regional folders.

Our folder directory relies on Next.js’s dynamic segments using […] nomenclature and looks something like this:

app
    [region]
        [locale]
            layout.tsx
            page.tsx
            // routes
            account/
                page.tsx
        layout.tsx // per‑region layout if needed

These route parameters are then exposed by utility functions to developers and passed to the BFF to allow for easy localization of features. For example in Hong Kong we want to display all prices in Hong Kong Dollars.

Note: when developing locally we can also rely on any sub-domains of localhost resolving to 127.0.0.1 meaning we have no environment specific logic and simple set up. E.g. tw.localhost:port/en

Linking regions

Finally, given we now have multiple domains, we need to be careful to avoid internal competition with pages from multiple domains being indexed and competing against each other. We do this by adding metadata to the pages HTML that bots can parse to infer that each page has multiple localized variants. First we define the lang in html element corresponding to the combined region and language,

<html lang="en-TW" dir="ltr">

Through this, bots will understand that this content is intended for users residing in Taiwan who have a preference for English. When typing in a search, your search engine will typically use a combination of factors such as your IP Address, browser preferred language etc to present you with the best possible results.

Additionally we add alternate links for all other variations of the page:

<link rel="alternate" href="https://tw.mercari.com/en" hreflang="en-TW"/>
<link rel="alternate" href="https://tw.mercari.com/zh-hant" hreflang="zh-Hant-TW"/>
<link rel="alternate" href="https://hk.mercari.com/en" hreflang="en-HK"/>
<link rel="alternate" href="https://hk.mercari.com/zh-hant" hreflang="zh-Hant-HK"/>

This further signals to the bot that each regional sub-domain is intended for users of that specific region and prevents, for example, Taiwanese pages showing up for Hong Kong users when searching etc.

Moving forward

We have created a solid base and in the coming months will be working on removing the remaining feature gap with the Japan Marketplace application to ensure optimum UX for our users in addition to optimizing the application for improved performance rollout to multiple regions in the next year.

If you’re interested in web technologies please check back later in this series where I will be discussing how we have used modularization to enable greater shareability of our frontend code and specifically how we developed a new library to stitch the i18n resources of these modules together for an application.

Thanks for reading and please check back again tomorrow for Ryuyama’s article

E2E Tests Every Developer Can Write — Test Platform Built with Plain go test

Fri, 17 Oct 2025 11:19:28 GMT

Introduction

Hi! My name is @ryotarai, and I’m responsible for SRE & Enabling for Crossborder (XB) Engineering.

As part of the series, Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App,” this post takes a deep dive into end-to-end (E2E) testing for the project’s backend APIs. Specifically, I’ll share how we built an E2E testing foundation that any developer can maintain, and I’ll cover the design philosophy and its implementation.

Why We Needed to Improve E2E Tests

Challenges with conventional E2E testing

E2E tests for backend APIs play a crucial role in verifying that the entire system functions correctly. Despite this, many projects run into the following problems.

Complex setup: Preparing the test environment takes time, which does not allow developers to run tests readily.
Hard to run test in parallel: Test must compete for resources, leading to long runtimes.
Reliance on individuals: The QA team is the principal organization in charge of maintaining tests, which makes it hard for developers to work with tests themselves.
High learning cost: Testers have to learn how to use specialized frameworks or DSLs.

At the outset, our project faced these issues too. Especially when only the QA team maintained the E2E tests, we ran into a number of problems like the following:

API changes pushed E2E test updates down the priority list.
Slow test additions led to lower coverage.
Developers did not understand test implementations, complicating debugging.

Our Goal: E2E tests that allow everyone to contribute

Our goal was a structure that allowed every developer writing API code to maintain the E2E tests, instead of just the QA team.

To make that possible, the setup needed to meet the following requirements:

Be able to write tests using technologies developers already use daily
Ensure a low learning cost so testers can get to work immediately
Be able to use IDE features like code completion and refactoring
Be able to run the same way locally and in CI

Framework Design Philosophy

The philosophy: “Write it with plain `go test`”

Mercari Global App backend APIs are implemented in Go. Ultimately, we chose to write E2E tests as ordinary Go code using go test. There were a few reasons for this:

Zero learning cost: Developers already know how to write code in go test.
Type safety: Developers can directly use Connect’s generated clients and get compile-time checks.
IDE support: Completion, refactoring, go-to-definition, and more are all available.
Easy debugging: Team members can debug it like any regular Go program.
Leverage existing code: Test helpers, mocks, etc. can be reused.

This move changed E2E tests from something “apart” from our work into a part of our everyday development workflow.

At the core of our E2E framework is the design principle: “You can write it with plain go test.”

Let’s look at a real test example:

func TestUpdateNickname(t *testing.T) {
    t.Parallel()

    tests := []struct {
        name     string
        userID   int64
        nickname string
        wantCode connect.Code
    }{
        {
            name:     "Success",
            userID:   createTestUser(t).ID,
            nickname: "NewNickname",
            wantCode: connect.CodeOK,
        },
        {
            name:     "Blank nickname returns error",
            userID:   readonlyUser().ID,
            nickname: "",
            wantCode: connect.CodeInvalidArgument,
        },
        {
            name:     "Non-logged in user returns error",
            userID:   0,
            nickname: "TestNickname",
            wantCode: connect.CodeUnauthenticated,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            t.Parallel()

            testenv.Run(t, func(params env.RunParams) {
                client := accountv1connect.NewBFFAccountServiceClient(
                    http.DefaultClient,
                    params.Server.URL,
                )

                req := connect.NewRequest(&accountv1.UpdateNicknameRequest{
                    Nickname: tt.nickname,
                })

                if tt.userID != 0 {
                    // Set authentication header
                    setAuthHeader(t.Context(), req.Header(), tt.userID)
                }

                _, err := client.UpdateNickname(t.Context(), req)
                if connect.CodeOf(err) != tt.wantCode {
                    t.Errorf("error code = %v, want %v",
                        connect.CodeOf(err), tt.wantCode)
                }
            })
        })
    }
}

This code uses Go’s standard table-driven test pattern:

Use t.Parallel() to specify enable parallel execution (same as regular go test)
Define test cases in a slice of structs
Use t.Run() for subtests; each subtest also runs in parallel
Inside testenv.Run(), obtain the test server URL
Use Connect’s auto-generated client as-is
Use the same assertions as the regular go test

There’s almost no complexity related specifically to E2E, you simply write the test like you would a regular unit test.

Additionally, because you can write in plain Go code, you can also effectively leverage AI coding tools like Claude Code. With AI assistance, you can add test cases and flush out edge cases more efficiently. Even for team members outside backend engineering (like QA) who aren’t yet accustomed to Go, AI helps them author their test code.

We also leaned heavily on AI when migrating existing E2E tests implemented in Jest to this framework. We managed to make the migration efficient by referencing the existing tests, having AI generate Go test code, and then having developers review and tweak it.

Overall architecture

One option for running E2E tests is to point them at an app deployed in a shared development environment accessible to everyone. However, there is an issue with this approach, namely that it makes it hard to test in-progress backend changes immediately.

We prioritized having an environment where we could run E2E tests while changing application code, and add or modify tests on the fly. To achieve that, we adopted a design where we dynamically started a server for each test. This allowed developers to validate their changes with E2E tests immediately and even do test-driven development.

Main responsibilities of the framework:

Automatic startup and management of test servers: Start servers on demand and manage them in a pool.
Automatic database preparation: Start AlloyDB Omni, create logical databases, and run migrations.
Parallel-execution support: Manage resources so multiple tests can run concurrently.
Automatic cleanup: On test completion, automatically clean up data and return resources to the pool.

From a developer’s perspective, all this complexity is fully hidden. Just call testenv.Run() and your test environment is ready.

Implementation Details

Next, let’s take a look at the implementation of these ideas to see how the framework achieves parallel execution and resource management.

Parallel execution via resource pools

To enable parallel E2E execution, we manage servers with a pool.

Crucially, when the function passed to testenv.Run() returns, the server is automatically returned to the pool. Developers don’t need to manually release resources. They simply write tests as usual and the framework handles cleanup and pooling.

This setup provides the following:

No resource contention during parallel runs
Minimized server startup cost (reuse from the pool)
Prevention of data contamination between tests (initialize with TRUNCATE)
Transparent resource management (developers don’t need to think about it)

Database management

For the database, we start with only one AlloyDB Omni container. Inside the container, the framework automatically creates a logical database for each test and runs migrations.

This design provides the following:

Reduced startup cost (only one DB container has to start)
Data isolation even under parallel execution (each logical DB is independent)
Automated migration (developers don’t have to think about this

Logical databases are also managed with a pool. After a test, we truncate to clean the data and then reuse the database.

Collecting code coverage

The framework supports go build -cover, introduced in Go 1.20+.

Ordinary test coverage (go test -cover) only measures execution within test code, but E2E needs to measure the server process itself. This is what go build -cover enables.

Our framework implementation covers the following:

Automatic creation of an independent coverage directory per server
- Create a temp directory on each server startup
- Automatically set the GOCOVERDIR environment variable
Accurate coverage collection even with parallel execution
- Each server writes to its own directory, so there are no conflicts
Automatic merge when test ends
- Consolidate all server coverage data with go tool covdata merge
- Produce a single, consolidated coverage dataset

Developers only need to set specific environment variables to automatically collect and merge coverage across multiple servers:

# Build the server binary with coverage
go build -cover -o server ./server

# Run tests while collecting coverage
GLOBAL_GOCOVERDIR=/tmp/coverage go test ./e2etest/...

# Generate a coverage report
go tool covdata percent -i /tmp/coverage

This setup enables accurate code coverage for E2E tests, helping to quantify API quality.

Running on Kubernetes

We run E2E tests locally during development and on Kubernetes in CI. Here are some interesting tricks for running on Kubernetes.

Fast deployment with `go test -c`

Here are some of the tasks you might typically do to run tests on Kubernetes:

Build a container image
Push the image to a registry
Pull the image in a Kubernetes Pod
Start the container

However, each of these steps takes time to complete. Since speed matters for E2E, we took a different approach:

# Build the test binary
go test -c \
    -o package/e2etest \
    ./path/to/e2etest

# Build the server binary
go build \
    -o package/server \
    ./path/to/server

# Archive with tar and transfer via kubectl exec
tar -czf - -C ./package . | \
    kubectl exec -c main -i -n ${POD_NAMESPACE} ${POD_NAME} -- \
    tar xzf - -C /tmp/e2e

# Run directly inside the Pod
kubectl exec -c main -it -n ${POD_NAMESPACE} ${POD_NAME} -- \
    /path/to/entrypoint.sh

By using go test -c, you can compile tests into an executable binary. That translates into three things::

No need to build a container image
No pushing/pulling from a registry
Direct file transfer via kubectl exec

Using this method, we cut the lead time to start running tests significantly; from build to test start takes about a minute and a half.

We run on Kubernetes to secure enough resources for parallel execution. As you increase the degree of parallelism, you need that many servers to avoid pitting tests against each other, so resource needs grow linearly—hence the cluster.

Conclusion

In this post, we introduced our E2E testing approach for the Mercari Global App backend APIs. With this approach, E2E tests are no longer “apart” from our work and are instead a part of our everyday development flow. Now, when developers change APIs, they can add or modify E2E tests freely.

Of course, there’s still room to improve:

Further reduce test execution time
- We’re working on running only the tests relevant to the changes using AI
Simplify test data setup
Improve test result reporting

Still, by prioritizing the developer experience, we believe we’ve built a sustainable E2E testing foundation.

We hope sharing our work will be helpful to projects grappling with similar challenges.

Toward a Global Identity Platform

Tue, 14 Oct 2025 18:14:37 GMT

Toward a Global Identity Platform

Introduction

Hi! I’m gia, from the Mercari ID Platform team. Our team is in charge of authentication and authorization across Mercari group services.
This article is part of the blog series Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”.

In this article, I’d like to share how we extended Mercari’s Identity Platform to support global accounts, contributing to the company’s ongoing global expansion efforts.

The global expansion initiative

Starting point

Mercari’s cross-border business began in 2019 through collaborations with international partners. In this model, Mercari items are listed on partner platforms, allowing overseas users to purchase them. However, there is no direct interaction between these users and Mercari’s services. Instead, transactions are facilitated by proxy buyers, who purchase items from the Mercari C2C marketplace and then ship them to the end users. While this approach enables international access, it also limits the overall user experience — for instance, users are unable to use coupons, participate in promotional campaigns, or access certain platform features.

Until 2024, Mercari’s Identity Platform exclusively supported Japanese users. The platform functions as an identity provider (IDaaS) for all Mercari Group services, offering robust authentication, authorization, and access control capabilities. It supports a variety of authentication methods, including passwords, SMS one-time passwords (OTP), social network (SNS) logins, and Passkeys. Through this system, users can create a Mercari account and use single sign-on (SSO) to access Mercari’s services seamlessly. Our clients range from web-based applications to mobile apps, though the IdP frontend was standardized to provide a consistent web-based experience across platforms.

Over the years, our account system has evolved alongside Mercari’s growth. However, a significant portion of it still resides within a legacy PHP monolith. Although we have made substantial progress toward migrating to a microservices architecture, the transition is still ongoing. Decoupling the account system is a crucial step in strengthening and modernizing our Identity Platform to better support Mercari’s global ambitions.

Initial requirements

Our team learned about Mercari’s global expansion initiative at the beginning of 2024. The first milestone focused on enabling Taiwanese users to create accounts and purchase items directly from the Mercari C2C web platform. At this initial stage, users would have access only to marketplace features, while other offerings—such as Fintech services—were intentionally excluded.

Because this effort represents the acquisition phase, we prioritized making account creation as simple as possible. To achieve this, we initially adopted a straightforward email-and-password registration flow. Given that users interact with Mercari exclusively through the web interface, verifying their email addresses was essential to ensure that important notifications could be reliably delivered.

Additionally, supporting multiple languages was a key requirement. A significant amount of content—including the Terms of Service, Privacy Policy, and other user-facing materials—needed to be properly localized to provide a smooth and trustworthy experience for our new users.

Global Identity Platform Challenges

When a business expands into new countries/regions, it inevitably encounters unique challenges depending on the region, business model, and underlying systems. In our case, several key challenges stood out during the early phase of global expansion:

1. Legacy System Dependency

As described earlier, our account system still relies heavily on legacy components. Migrating existing Japanese accounts is not a simple task, and introducing global accounts presented an additional layer of complexity. Since global accounts were built from scratch, we wanted to avoid incorporating them into the old system. However, many existing services still depend on the legacy account infrastructure, making it a significant challenge to decouple the systems while ensuring overall service continuity.

2. Access Control

Not all Mercari features are available to global users, which means feature access must be carefully controlled based on a user’s region. Additionally, the initial global rollout uses only email-and-password authentication, which doesn’t have a high assurance level. Features that require stronger assurance must therefore be protected by mechanisms based on Authentication Assurance Level (AAL) and Identity Assurance Level (IAL).

3. Internationalization and Localization

Expanding from a Japan-only platform to a multilingual system is far from trivial. Much of our existing UI and business logic was tightly coupled with Japanese-specific design and assumptions. Designing an extensible approach for adding new languages—both for the frontend and backend—proved to be a significant technical challenge.

4. Different Signup Requirements per Region

Each country/region introduces its own regulatory and compliance requirements. For example, some regions mandate age verification during account creation, while others require distinct KYC (Know Your Customer) procedures. Designing a flexible yet maintainable system to accommodate these diverse regional requirements—without introducing excessive complexity—was another critical challenge.

Global account registration

Legacy system decoupling

To support global accounts, we developed a new registration flow encompassing both backend and frontend systems. In this new flow, account data is no longer stored in the legacy system; instead, it resides in a dedicated microservice. However, because many existing services still reference the legacy database, we needed to maintain backward compatibility. To achieve this, we implemented a reconciler that synchronizes data between the new and legacy databases, ensuring consistency across systems.

Each user now has a single account that can be used globally. To support this, we introduced a country_region_code attribute to the account model, enabling regional differentiation and handling for various use cases.

Account readiness check

Typically, account registration involves multiple steps spanning different business domains. For example, email-and-password registration falls under the Identity team’s ownership, while KYC (Know Your Customer) processes are managed by the KYC team. In the legacy system, where a single database and API server handled all processes, temporary data could be stored in intermediary tables before being finalized.

In contrast, the new microservices architecture separates responsibilities by domain, with each service managing its own database. This made the use of temporary tables impractical. To address this, we built an Account Signup Orchestrator—a service responsible for validating account readiness and coordinating the various signup steps across domains. It also enables a more resilient user experience: if a user drops out partway through the registration process, they can resume from where they left off without starting over.

Customizable registration flow

As mentioned earlier, account signup requirements can vary significantly by region. If the registration flow logic were to be managed entirely on the frontend, maintaining and updating these conditions would quickly become complex and error-prone. To address this, we decided to centralize condition management on the server side while keeping the frontend as an orchestrator.

In this approach, the frontend orchestrator retrieves signup instructions from the server, dynamically rendering the appropriate user interface based on those instructions. It continues this process iteratively—fetching new instructions and updating the UI—until it receives a signal indicating that the flow has been completed.

This architecture not only simplifies how we handle region-specific signup conditions but also allows us to manage different authentication requirements, such as the elevation flow (described in the next section), in a consistent and extensible way.

Global account login

Ideally, we aim to provide a single, unified login page that serves both Japanese and international users. However, the available authentication methods must be tailored by region. For instance, LINE login should only appear in regions where the service is supported. Due to time constraints during the initial rollout, this unification was not implemented, and we currently maintain separate login pages for non-Japanese users. Nevertheless, we are actively working toward consolidating these experiences to deliver a more seamless and consistent login flow for all users.

Region based access control

To manage feature availability across different regions, we introduced a region-based access control mechanism, implemented at two layers: the authorization server and the resource servers.

Authorization server

At the authorization server level, access control is enforced during the token issuance process. We extended the OAuth 2.0/OIDC client settings to include a list of supported countries/regions.

ClientID	Supported countries/regions
ClientID_A	JP
ClientID_B	HK, TW

When a client requests an access token, the authorization server verifies that the account’s country_region_code is included in the client’s supported region list before issuing the token. This ensures that only clients operating in approved regions can obtain valid credentials.

Resource server

Once an access token is issued, clients can interact with various resource server APIs. However, not all endpoints are globally available. Each resource server owner can define which regions are supported for specific endpoints through configuration. During token verification, the resource server checks whether the country_region_code associated with the access token is permitted for the requested endpoint. If the region is not allowed, access to that endpoint is denied.

Together, these two layers of validation provide a robust and flexible framework for controlling feature access by region while maintaining consistency across Mercari’s distributed system architecture.

Internationalization and localization

As you can imagine, supporting global accounts required us to handle a large amount of content that needed both internationalization (i18n) and localization (l10n). While internationalization focuses on presenting the same content in multiple languages, localization ensures that content is appropriately adapted for each specific region or culture.

To support our global rollout, we added new versions of key materials such as the Terms of Service, Privacy Policy, and email templates, along with translations for a wide range of user-facing messages across our systems.

We leveraged the ui_locales parameter defined in the OpenID Connect specification to manage language selection dynamically. Users can also specify their preferred language, allowing us to deliver a more personalized and accessible experience across regions.

Global phone number support and elevation flow

To enable users to access features that require multi-factor authentication (MFA)—such as coupons and promotional campaigns—we needed to introduce global phone number support. Unlike the Japanese C2C marketplace, where phone numbers are required at signup, we chose not to request them for global accounts to avoid forcing users to complete two separate OTP verifications (one for email and one for phone number) during registration.

Instead, phone number registration is initiated dynamically through an AAL/IAL-based access control mechanism. Each API endpoint is pre-configured with a minimum required level of assurance. When a user attempts to access a protected endpoint, the system verifies whether their account and current authentication session meet the required assurance level. If they do not, the request is rejected, and the client triggers a flow prompting the user to provide additional information or complete further authentication steps. We refer to this as the elevation flow.

The elevation flow supports multiple verification types—not only phone number verification but also additional methods such as Passkey registration and authentication challenges. It was designed based on the principles of the Step-Up Authentication Challenge Protocol, leveraging the acr_values and claims parameters to determine which verification method should be initiated. These requirements can be specified by the client or enforced directly by the authorization server through configurable policies.

This approach gives us a flexible and secure way to progressively elevate user assurance levels, improving both security and user experience without overcomplicating the initial signup process.

Global mobile application support

After successfully rolling out our Marketplace web services to additional regions, we began developing a global mobile application to deliver a more seamless and localized experience for our international users. Similar to the Japanese app, the IDP flows are handled within in-app browsers. To enable session sharing between the in-app browser and external browsers—allowing users to seamlessly single sign-on (SSO) into web services—we utilized ASWebAuthenticationSession on iOS and Chrome Custom Tabs on Android.

The global app supports multiple regions and includes pre-login features. Upon the first launch, users are asked to select their country/region. This introduced a unique challenge: the country/region selected in the app might differ from the country_region_code associated with the user’s account after login. For new registrations, we can safely skip the country/region selection step during signup. However, in the case of sign-ins (or when users are already logged in through web services), the system must detect and handle mismatches between the selected country/region and the account’s country_region_code. To resolve this, we implemented an error-handling flow that returns a response to the application side, prompting a country/region change process within the app.

As part of this initiative, we also took the opportunity to standardize our logout process to align with the RP-Initiated Logout specification, ensuring consistency across platforms and compliance with OpenID Connect standards.

Future works

As we continue expanding Mercari’s global presence, several key initiatives are underway to support our next phase of growth:

1. Automating Country/Region Rollouts

To accelerate our global expansion, we aim to automate the process of launching Mercari services in new countries/regions. While AI can play a valuable role in streamlining these rollouts, our immediate focus is on simplifying and standardizing the underlying processes to ensure scalability and reliability.

2. Centralized Account Management Portal

We have begun an initiative to develop a unified user account management portal. Currently, each Mercari service maintains its own account settings page, which can lead to inconsistency and confusion. By consolidating these into a single, centralized portal, we aim to provide a more cohesive and user-friendly experience across all services.

3. Expanding Passwordless Authentication

Our passwordless authentication initiative has been active in Japan for some time, and we’ve learned many valuable lessons from that experience. We plan to extend this capability to global accounts, offering a faster, more secure, and frictionless sign-in experience for international users.

4. Regulatory Compliance and Regional Expansion

When expanding our business to other countries/regions, ensuring compliance with regional regulations (e.g. GDPR, CCPA, COPPA) is essential. Strengthening our privacy and data protection frameworks will be a key step in achieving this.

5. Advancing Global eKYC and Digital Identity

Finally, we are exploring enhancements to our global electronic Know Your Customer (eKYC) procedures. Leveraging digital wallets and verifiable credentials presents an exciting opportunity to streamline identity verification, similar to how we have successfully integrated Japan’s My Number system.

These efforts reflect our ongoing commitment to building a secure, scalable, and globally accessible identity platform that empowers users around the world to engage with Mercari seamlessly.

Finally

On November 13, 2025, Mercari Group’s tech conference, Mercari GEARS 2025, will take place. I’ll be presenting a poster session on Mercari’s Global Identity Platform, where I’ll share more insights and discuss our ongoing challenges and future plans.

If you have any questions or would like to chat after reading this article, please feel free to stop by and talk with me at the event. There will also be many other exciting sessions covering a wide range of topics—be sure to check them out!

Tomorrow’s article is by @Karthi. Please continue enjoying the Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”.

Behind the Scenes of SRE Supporting the Global Web — An Improvement Approach to Accelerate Development

Mon, 13 Oct 2025 10:00:04 GMT

I’m hatappi, working on SRE & Enabling at Cross Border (XB) Engineering. In addition to our SRE role, our team also serves as an Enabling team as defined in Team Topologies, supporting (enabling) XB developers to deliver value more smoothly through technical problem-solving and environment optimization.

In July 2025, I transferred from the Platform Network team to the XB SRE & Enabling team, where my first assignment was working on the launch of Mercari Global Web. This article, as part of the Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”, focuses on the Global Web being developed alongside the app, and introduces the approach I’m practicing to deliver value as an SRE in this new environment.

Approach to Problem Discovery for Enabling

When I transferred to the XB SRE & Enabling team in July, my first mission was "enable the launch of the Global Web." However, having just transferred, I didn’t know what challenges or improvement points existed. I believed that correctly understanding the current situation was essential, so I took two approaches.

1. Trying It Myself

To understand and empathize with the challenges Global Web members face, the quickest way is to experience it myself. Therefore I tackled one feature development task. This allowed me to experience not just setting up the local environment and checking operations, but the entire development cycle from implementation, creating a pull request, receiving reviews, to merging.

Through this approach, I identified various improvement points including CI execution requiring long wait times for feedback and local dev server startup being slow.

2. Listen to Voices

Relying solely on your own experiences inevitably narrows your perspective. It’s particularly important to hear from members who develop the Global Web as their daily work. I gathered information by asking about improvement points on Slack and participating in planning and retrospective meetings.

This not only identified issues I couldn’t discover alone but also helped me in prioritizing them. For example, while I identified slow CI execution time as an improvement point, in reality CI execution time only became concerning at the final stage when it was necessary to receive reviews from other members. The instability of CI caused by occasional failures and slow local server startup times were perceived as bigger challenges.

Insights Gained from Platform Engineering Experience

While implementing these two approaches, I had the opportunity to reflect on my previous experiences with the Platform Network team.

As part of the Platform Network team, we provided shared infrastructure and tools that could be horizontally deployed across multiple Mercari products as part of Platform Engineering. Mercari has multiple products, each with its own unique context and domain knowledge. This presented a challenge, as it was difficult for the Platform side to deeply immerse itself in and fully understand the specifics of every product’s environment.

Through the implementation of the two approaches as a member of XB’s SRE & Enabling team, I have come to re-recognize the importance of deeply engaging with the field. At the same time, my experience in Platform Engineering has also helped me understand the significance of horizontal deployment when considering Mercari as a whole.

At Mercari, there are still not many engineers who have experience in both areas. That is precisely why I am actively providing feedback to the Platform team based on the experiences I have gained at XB, such as the recent improvements related to the global web, and working together to drive improvements forward.

Problem Solving Utilizing AI

In the previous section, those two approaches greatly increased the resolution of issues to tackle. However, having just transferred, knowing what to improve was not enough. I still needed to catch up on much information including Global Web and Cross Border contexts, as well as Web-related technologies. To smoothly enable the Global Web launch, I attempted to streamline this process by leveraging AI.

Learning and Research

For example, let’s say we’re tackling the issue of slow CI execution time. To understand this, I first need to understand how it works. As explained below, I considered necessary information and improvements while utilizing Claude and Claude Code.

First, I used Claude Code to investigate existing CI-related configurations. Since Mercari uses GitHub Actions, I asked about Action purposes and confirmed dependencies between Jobs while reading the source code to deepen my understanding. During the research, I encountered technologies I wasn’t familiar with, such as Turborepo.

When encountering unfamiliar technologies, I read the official documentation as the primary source to understand them. I utilized Claude for summarizing documentation. While Claude Code could be used directly for this, I chose Claude for its Artifacts feature (Fig1). Artifacts is a feature for creating independent content that can be created and edited during conversations with Claude. This allows me to deep-dive into unfamiliar technologies while creating comprehensive documentation that’s easy to reference later.

Fig1: Claude Artifacts

The final step was research for improvements. As a first step in considering improvement methods, I utilized Claude Research to collect general improvement strategies. For example, "Research common methods to speed up CI in the repository using Turborepo." This allowed me to efficiently list multiple approaches such as improving cache strategies and optimizing parallel execution in a short time, enabling efficient hypothesis formation for improvements. Additionally, using Artifacts, I could compile information toward final implementation based on the research findings.

Implementation and Review

Once improvement hypotheses are established, the next step is implementation. The information compiled during research exists in Artifacts and can be output as Markdown, which can be used with any tool or model such as Claude, GPT, or Gemini.

I primarily use Claude Code because of Slash commands. Slash commands are special commands starting with / in Claude Code that can execute specific operations. I’ve migrated processes I perform during development to these Slash commands. For example, there’s a Slash command for creating pull requests from changes. This Slash command defines not just pull request creation but also steps I frequently perform, such as considering commit messages from changes and committing.

After implementation comes review. While I have Claude Code review as well, I also perform reviews myself. Previously, I used git diff or diff viewers attached to editors. However, I often found improvement points when creating pull requests on GitHub that I thought were fine during local review. Making changes and pushing every time to check on the pull request takes time. To solve this problem, I started using difit.

difit is a CLI that provides GitHub-like views in the local environment (Fig2). As it’s added as an npm package, installation is simple and you can start using it immediately. With GitHub-like views, I can now do locally what I used to do on pull requests. Additionally, difit has a comment feature with copy functionality that allows added comments to be passed as prompts to AI. Thanks to this, the cycle of developing with Claude Code while reviewing and improving can now be completed locally.

Fig2: difit (https://github.com/yoshiko-pg/difit)

Debugging

Finally, debugging. I usually use Chrome. Chrome DevTools is indispensable for debugging. However, with its various features, I always struggled with which features to use and where to look to find the information I needed.

Therefore I tried the recently released Chrome DevTools MCP. This feature extracts necessary information by operating DevTools through an MCP Server with just natural language instructions. For example, just entering "Check the performance of this Global Web page" analyzes relevant metrics.

This allowed me to smoothly perform DevTools operations that I previously struggled with, reducing the time to problem discovery.

Learnings from Enabling Activities

I learned two things through this Global Web enabling experience.

The Importance of Entering the Field and Engaging with Primary Information

The first is the importance of being in the frontline and accessing primary information. If I had made judgments based only on objective metrics, it would have been difficult to notice that occasional CI instability and local server startup time were bigger problems than the CI execution time that development members were feeling.

The Effectiveness of AI When Challenging New Technology Areas

The second is that AI lowers the barriers when challenging new technology areas.

Even understanding the importance of the first approach, hesitation occurs if the hurdle to practice is high. However, I felt that utilizing AI made it easier to overcome this "initial wall." Of course, AI doesn’t solve everything, but I feel that my preferred approach of first creating something that works and then deeply understanding its mechanisms can now be done more smoothly.

Future Work

As mentioned in Rebuilding App and Foundation for Global Expansion, expansion to 50 countries and regions is planned within the next three years, which is technically very challenging. To expand globally at this speed, there are many things to consider: what implementation and settings are needed, how we can optimize efficiency, where to place data, where to locate web servers, and how we can utilize CDN. Because there’s so much to consider, it’s interesting and I feel it’s where SRE & Enabling can really shine, so I’m very excited about it.

Conclusion

This article introduced how I’m advancing Global Web Enabling in a new environment after transferring from the Platform Network team.

On November 13, 2025, Mercari Group’s tech conference "Mercari GEARS 2025" will be held. I’ll be talking about the CDN migration I worked on when I was in the Platform Network team. There are many other interesting sessions, so please join us!

Tomorrow’s article is by @gia. Please continue enjoying the Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”.

The Journey of User-Generated Content Translation

Sun, 12 Oct 2025 09:00:03 GMT

This is @aymeric from Cross Border Engineering.

This article is part of the blog series Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”

Introduction

The Mercari Global App represents the latest significant milestone in Mercari’s ongoing global expansion strategy. However, the translation of user-generated content, such as product listings on Mercari, predates this, commencing in October 2023.

Implementing translation capabilities led to a measurable increase in transactions by several percentage points, as demonstrated by A/B tests. This improvement occurred despite the availability of native browser translation features, highlighting the significance of integrated translation for user experience.

Over two years, the cost of translation dropped by 100x. Translation now costs Mercari 1% of what it cost two years ago, thanks to sharp decreases in Large Language Model (LLM) pricing.

This initiative initially aimed to boost sales of Mercari products through proxy partners exporting from Japan. It later evolved to support Mercari’s direct expansion into Taiwan in August 2024, eventually integrating with the Mercari Global Product.

DeepL and Google Translate offer classic translation services with pay-as-you-go APIs, high rate limits, low latency, and pricing based on the number of input characters. These services support a wide array of languages, covering all countries Mercari aims to expand into, and provide glossary support. This ensures consistent translation of specific terms, such as "メルカリ" to "Mercari" or "カビゴン" to "Snorlax," preventing phonetic translations like "Kabigon".

In contrast, LLM API pricing is based on input and output tokens, with differing costs and stricter rate limits for pay-as-you-go APIs, and higher response times. The input prompt significantly influences results and contributes to the overall input token cost. Language support is often vaguely documented, leading to occasional confusion between similar languages like Traditional and Simplified Chinese. Furthermore, LLMs currently lack glossary support and new models are released regularly while other models get deprecated.

This article recounts the trials and tribulations of this journey, from using a classic translation model to leveraging LLMs, the problems that we faced, and includes translation-related non-AI features.

Static content vs. user-generated content

To understand the complexities of item translation, it’s crucial to distinguish between static and user-generated content.

The image above is a Mercari product page. The title and description are “user-generated content”. The text was written by the user when they listed the item for sale.

Conversely, all other text elements on a product page—such as menus, section titles, button labels, and breadcrumb categories—constitute static content. These strings are created by Mercari and are stored directly within the codebase.

Other user-generated content includes the users’ profiles, comments on products, transaction messages, and user reviews after transactions end.

This article focuses specifically on the user-generated content.

The importance of understanding the product and users

In a Business-to-Consumer (B2C) model, a single product can be sold multiple times, allowing translation costs to be amortized across numerous transactions.

However, in a Consumer-to-Consumer (C2C) marketplace like Mercari, each product listing is unique. Consequently, the translation cost per transaction increases linearly with the volume of items listed.

Mercari covers both B2C and C2C models, yet the C2C inventory is a much larger volume than B2C.

As we started this initiative within Mercari Japan, all products are initially listed in Japanese then translated into multiple target languages. Currently, Mercari primarily focuses on English and Traditional Chinese translations, with plans to support additional languages in the future.

We considered and tested several strategies for translations:

Translating all products when they are created
Translating when users visit the product detail page
Translating when users tap a button
A mix of the above

This video presents the experience of a user visiting a product page that has never been translated before.

Translations are cached, eliminating the need for real-time translation during every user visit and improving page loading times for the next visits, as can be observed in the next video

Initially, we considered translating all products at creation and storing these translations for quick retrieval. However, this approach faced challenges due to the need to support multiple languages and the fact that a significant number of products are never viewed by our smaller international user base.

Translating only when a user visits a product detail page introduces latency for the first visitor but is the most cost-effective solution. We experimented with a "translate" button on product pages, but low usage and declining metrics led us to abandon this option.

Product updates also presented a challenge. A small fraction of users frequently update their products, sometimes to manipulate search rankings. If all updates were translated, this behavior, by less than 1% of users, would increase translation costs by 25%. To mitigate this, we implemented workarounds: updates are time-boxed, with only the last update within a window being translated. Additionally, small updates, based on character count, are not translated.

This approach occasionally leads to customer issues, such as a customer receiving a garment of the wrong size because a one-character size update went untranslated. However, reimbursing customers for these rare occurrences is more cost-effective than investing weeks of engineering and API costs to eliminate them entirely.

We always provide the original text, allowing users to switch between translated and original content, a feature we emphasize to our users.

Marketing efforts, beneficial for product sales, introduce additional translation requirements. Traditional marketing platforms like Google Shopping, Google Ads, and Meta Ads have varying levels of built-in translation support. This necessitates translating products at creation for marketing purposes. Fortunately, marketing teams prioritize ROI, and are willing to cover the translation costs for relevant product categories within their budgets 😀.

The technical implementation details will be discussed in the next section.

Recounting the technical iterations

Integrating a classic translation model

We decided to use DeepL for our initial translation model. This involved:

Automatically translating items when a user lands on the product detail page.
Storing these translations for future use.

This approach led to a statistically significant 5.3% increase in buy tap rate.

We considered using ChatGPT and GPT-3, which had recently been released, but ultimately chose a reliable service known for its high-quality Japanese translations. LLMs API pricing at the time was pretty high, so there was no strong upside going with an LLM solution.

The DeepL public pricing of 25$ per million input characters has stayed constant over this period.

Our first LLM

The decreasing cost of LLM API pricing motivated our transition to an LLM-based translation solution. This move offered potential cost savings and valuable experience in deploying LLMs in a production environment. We went with GPT-3.5 Turbo-0125.

To manage costs effectively, and considering any prompt would be counted as input token, we developed a concise and straightforward prompt for the feature:

* Original text will be delimited by ###\
* Original text is in Japanese\
* Your task is to translate it to Traditional Chinese

###

<the product’s title or description>

This prompt proved effective for a considerable period and with various models, which we will discuss later.

At Mercari, product titles are limited to 40 characters and descriptions to 1000 characters, with averages of 25 and 300 respectively.

Initially, we aimed to provide a structured input containing both the title and description in the prompt and retrieve their translated versions from the output. However, this approach presented challenges. When users updated products, they often modified either the title or description, making it inefficient to always send both. This also necessitated a constant decision on whether to send the title, description, or both.

Upon testing, the results were inconsistent, and accurately and reliably extracting the translated title and description from mixed outputs proved difficult. Consequently, we ultimately decided to translate titles and descriptions separately.

The main issues noticed with this prompt and model were the translation of names, like anime character names like the pokemons, or celebrity names from famous Asian bands. It often translated the Japanese name to its phonetic form. カビゴン became Kabigon instead of Snorlax, which we could not be resolved without a glossary.

We had to drop the glossary we used with DeepL. Despite that, all metrics stayed flat in our A/B test, and cost decreased by ~20%.

While the prompt above can be easily bypassed, it has not presented an issue, as the original Japanese text is consistently displayed on listings by sellers in Japan.

How LLMs Scale

We leveraged Microsoft Azure to access OpenAI’s GPT models.

Due to initial low pay-as-you-go rate limits for LLMs, which were insufficient for our needs, we utilized Azure’s "Provisioned Throughput Units" (PTU). PTU offers pre-paid, reserved processing capacity on a monthly basis, with a minimum reservable unit of 50 PTU and scaling in multiples of 50.

Mercari’s user traffic fluctuates throughout the day in a predictable pattern: low activity at night, increasing in the morning, remaining relatively steady during the day, peaking in the evening, and then declining as the day ends.

When utilizing PTU (Provisioned Throughput Units), it’s essential to strike an optimal balance between the traffic covered by PTU and the traffic handled by pay-as-you-go capacity.

Over-provisioning PTU to manage traffic spikes can lead to significant wasted expenditure on unused capacity during periods of lower demand. Conversely, under-provisioning PTU will result in frequent encounters with pay-as-you-go rate limits, potentially disrupting service.

The diagram below illustrates this mechanism.

This blog post from Microsoft explains this in details very well.

The downside is that it makes the implementation more complex as the PTU deployment and pay-as-you-go deployment use different endpoints, and it requires the application to detect rate limitation errors on the PTU to decide to make requests to the pay-as-you-go endpoint.

New models, better pricing: Transitioning to GPT-4o mini

Over the past two years, the cost of LLM APIs sharply decreased. Regularly reviewing and switching models was key to cost savings.

The migration to GPT-4o mini was purely motivated by the cost improvements, decreasing the cost by 7x.

We didn’t modify the prompt and ran a quick A/B test that showed flat business metrics, guaranteeing this model could be used safely in production.

From GPT to Gemini

Mercari’s engineering teams primarily use Google Cloud Platform (GCP). Our initial work with Microsoft Azure for translation services, utilizing GPT-4o mini, introduced complexities due to the unfamiliar environment and the need to re-establish infrastructure as code, authentication, and other platform-related aspects.

As Gemini became available, we made the technical decision to transition to Gemini on GCP. At the time, GPT-4o mini and Gemini 1.5 Flash had comparable pricing.

Another great advantage of Gemini was its much higher rate limits on the pay-as-you-go API. This meant we did not need PTU anymore, or GSU as Google calls it, for Generative AI Scale Unit.
This would simplify the implementation.

By the time we prioritized this transition, Gemini 2.0 was announced. We still opted to A/B test using Gemini 1.5 Flash as the price was cheaper than the new Gemini 2.0 models.

The A/B test showed no significant difference in business metrics. Consequently, we deprecated the GPT-4o mini implementation, permanently discontinuing our use of Microsoft Azure for this service, and launched with Gemini 1.5 Flash.

The significant cost reduction was a pleasant surprise, potentially due to an initial underestimation of Gemini 1.5 Flash’s cost or a pricing update from Google with the release of Gemini 2.0.
This brought our total cost reduction to 100x compared to our initial implementation with DeepL.

Interestingly, Gemini 1.5 Flash is the only model we’ve encountered that prices by character rather than by tokens, unlike other large language models.

First model forced retirement

As already mentioned, new LLMs get released regularly. And model providers also deprecate models just as regularly.
Google documents the retired models in this page. So far, all models get deprecated a year after their release.

Gemini 1.5 was released in two stages, four months apart. We initially adopted the first release (001), and when it was slated for deprecation, we had the option to migrate to either Gemini 1.5 Flash 002 or Gemini 2.0 Flash Lite. Due to its lower cost, we opted for Gemini 1.5 Flash 002.

Given the tight deadline and the perceived similarity of the models, we decided against an A/B test to save time. This proved to be a misstep.

The diagram below illustrates the latency we observed within days, which negatively impacted the initial user experience for product pages undergoing their first translation.

After investigating with Google, and as many of their clients moved from 001 to 002, they also observed increased latency. As the model would soon get deprecated, all internal capacity was used for Gemini 2.0 models, and so we decided to move to Gemini 2.0 Flash Lite.

And another Gemini model

Gemini 2.0 Flash Lite was released without A/B testing. While this brought latency down, it never reached the levels achieved by Gemini 1.5 Lite 001.

We observed an immediate side effect: the majority of translations began with a statement from the model, such as "Here is the translation:".

This issue was quickly identified and resolved by modifying the initial prompt.
With the cost per token having dropped significantly, we could design a longer prompt and landed on the following:

You are a Japanese-to-English translation API.

1. **Task:** Translate the content of the user's  tag.
2. **Output:** Your entire response MUST be the result, wrapped in  tags. Add no other text.

<content of title or description>

On the plus side, it performed much better on character and celebrity names, though they still occasionally get mistranslated.

This model is currently used at Mercari for product translation. It is scheduled to be retired on February 25, 2026.

Translation experience is not just delegating to a LLM: Non-AI features

Automatically translate or let the user trigger it to save money?

From the start, we had decided users should always be able to see the original content. So, a button to switch between the original content and the translated content was provided.

In the initial release of the translation feature, content translation was automatically triggered when a user landed on the product detail page.
Curious about the opportunity to reduce the cost, we ran an A/B test where content was not translated, so the user had to tap the “show translation” button to trigger the translation.
The business metrics went down very slightly and we decided to keep the automatic translation trigger.

How we measure the user experience

Later, to better measure the user experience, beyond the business metrics, we decided to add a button to let users report issues in the translation.

The translation feature is designed for simplicity, requiring no additional user context. A client-side event is sent to and stored in the backend when the feature is used.

Initially, we were unsure about the usefulness of the feature due to the lack of context and whether users would engage with it meaningfully. To assess this, we conducted an A/B test. We sampled and reviewed reports, with the condition that the feature would be retained if over half of the reports were justified.

The results showed that users did not misuse the button, and most reports were deemed valid. This not only confirmed known issues like character and celebrity names but also brought to light some less frequent problems.

Based on these internal translation issue reports and the A/B test results, we compiled a list of known issues. This list then formed the basis for a simple offline evaluation method, allowing us to quickly and more effectively assess new translation models.

Resolving translation issues: Implementing the glossary

One of our latest developments involves implementing a glossary to address persistent issues with character and celebrity mistranslations.

Given thousands of glossary entries, passing the entire glossary as input to the Large Language Model (LLM) for every translation is impractical due to prohibitive costs and latency. Effectively using a glossary goes beyond simple substring replacement. For instance, サイ refers to a character in Naruto, while サイズ means "sizes". Longer sequences, like スポイルじいさん (Old Man Spoil from One Piece), also require consideration.

To accurately match words and sequences, we introduced tokenization, which can be resource-intensive. Fortunately, we leveraged our existing search system’s Japanese tokenizer. By combining the glossary with tokenization, we could precisely identify parts of the input text requiring proper translation.

Our initial strategy involved replacing matched glossary entries in the input text with their translated values and then sending this modified text to the LLM. For example:

Original text: サイはナルトの登場人物です
Intermediate text (after tokenization and replacement): Saiはナルトの登場人物です
After LLM translation: Sai is a character in Naruto

This approach proved effective for English. However, results for Traditional Chinese were significantly poorer. The primary challenge was the close similarity between Japanese and Traditional Chinese characters, which made it difficult for the LLM to distinguish between content that needed translation and content that should remain unchanged.

Consequently, for Traditional Chinese, we adopted a different strategy. Instead of text replacement, we provided the LLM with additional context in the prompt, specifically a list of key/value pairs to be used in the translation. This alternative method yielded significantly improved results.

Key takeaways and future work

Mercari’s journey in user-generated content translation highlights a commitment to Mercari’s values, driven by a deeply iterative approach, emphasis on user experience, and strategic model transitions.

Key to this success was balancing cost with user experience, understanding the unique challenges of a C2C marketplace, and integrating crucial non-AI features.

One important observation is that newer models have no impact on business metrics. Considering high-end models are over 10x the price of the cheaper models, it’s hard justifying using more high-end models.

While significant progress has been made, there remains room for improvement, particularly in achieving more accurate translations and reducing latency. Furthermore, the continuous evolution and eventual deprecation of LLM models necessitate ongoing adaptation to maintain optimal performance.

Additionally, more user-generated content will soon be translated, such as user profiles, user comments on products, and more.

Finally

Thank you for making it to the end.

Credits for the work go to Amit Raj Baral and Christophe Labonne.

On November 13, 2025, the Mercari Group tech conference "mercari GEARS 2025" will be held where I will be one of the speakers.

Please join us! Registration is here 👉 https://gears.mercari.com/

Tomorrow’s article is by @hatappi.
Please continue to enjoy Series: Behind the Scenes of Developing ‘Mercari Global App,’ Mercari’s First Universal App.

Order Management in Mercari Global Marketplace

Fri, 10 Oct 2025 10:00:23 GMT

Hi! I’m takady, a backend engineer at Cross Border (XB) Engineering. In this post, I’ll share how we designed and built a flexible order management system from the ground up.

Background

While the existing Mercari Japan’s marketplace has a mature order management system, it’s not easily expandable to global marketplace requirements because of the following challenges;

One state transition pattern: The existing system’s order lifecycle is coupled to a specific business flow. Adding/Removing steps impacts the large areas of the system.
Dependencies in DB level: Maintain the data consistency between core resources by sharing DB transactions.

Rather than retrofitting the existing system, we chose to build a new order management system.

Design Decisions That Enable Scale

As the business grows rapidly, business requirements will vary significantly. Making the order management system easier to expand for future use-cases is the key to long-term success.

Flexible Lifecycle of Order Items

State transitions of each item can be defined differently depending on the product type. This flexibility allows the product to easily handle different business scenarios.

Note: These flows are examples to illustrate what our flexible lifecycle system enables. Not all flows are currently implemented, but the architecture makes it straightforward to add new ones as business evolves.

How Lifecycles Work: Data Structure

Each order item references a lifecycle that defines its available state transitions:

This approach means:

Adding new transaction types only requires defining a new lifecycle without significant code changes
Multiple items in the same order can follow different lifecycles independently
State validation is enforced at the lifecycle level, preventing invalid transitions

Orchestrating Distributed Transactions

One of the biggest challenges in any order management system is coordinating actions across multiple services while maintaining data consistency.

Saga Pattern with Orchestration

We employ the Saga pattern with an orchestration approach to manage distributed transactions in comparison with the following options:

Saga vs TCC: We primarily use the Saga pattern, which rolls back completed operations through compensating transactions when a step fails. While TCC (Try-Confirm-Cancel) provides stronger consistency guarantees, it requires extra effort as all participant modules need to implement separate Try, Confirm, and Cancel APIs. We only consider TCC on a case-by-case basis for operations that are difficult to roll back.
Orchestration vs Choreography: We chose orchestration over choreography because it provides a centralized coordinator that manages all interactions between services. This gives us a holistic view of the system, making it simpler to implement, maintain, and troubleshoot compared to a distributed choreography approach.

Our in-house orchestration tool, “Magician” (developed by Merpay), handles this complexity elegantly. Magician is an orchestration engine for distributed transactions that provides essential functionalities for our use cases, including workflow management (Saga), retry activity, and async workers. A significant advantage is that it’s already proven in production—several microservices at Merpay and Mercoin use it for mission-critical payment orchestration. This means we can reference real-world implementations and leverage shared knowledge across teams.

Example orchestration flow for crossborder order placement:

Validate order
Consume coupon from the promotion module
Process payment through Merpay
Place order in the order module
Request proxy partner to purchase items from seller

If any step fails, Magician automatically triggers compensating transactions to roll back previous steps, ensuring eventual consistency across services without requiring distributed locks.

These technical foundations—flexible lifecycles and robust orchestration—work together to create a system that can evolve with business needs while maintaining reliability and consistency.

Why It Matters: Business Impact

These technical decisions directly translate to business value:

Faster Time-to-Market for New Features

New transaction types can be launched without modifying existing order flows
Teams can develop and test new features in isolation, reducing coordination overhead
Example: Adding a new luxury goods verification step doesn’t require regression testing all existing product types

Reduced Downtime and Improved Reliability

Orchestration with automatic rollback prevents partial failures from leaving the system in an inconsistent state
If payment processing fails, coupons are automatically restored—no manual intervention needed
Centralized coordination makes it easier to monitor and troubleshoot issues before they impact customers

Building Checkout and Payment with Merpay’s Solution

Beyond orchestration for order management, we also needed to build checkout and payment capabilities. Rather than building from scratch, we integrated with Merpay’s established checkout and payment foundation —allowing us to focus our engineering efforts on marketplace-specific features.

What Merpay’s Solution Provides

Merpay’s checkout solution provides the foundation for our checkout flow:

Flexible checkout UI: The element concept allows us to customize the checkout experience
Multiple payment methods to be supported: Currently credit cards are supported, more payment methods to be supported in the future
Provider abstraction: Seamless integration with Merpay payment service while encapsulating the actual payment providers behind it

This integration enabled us to:

Accelerate time-to-market: Launch checkout capabilities quickly while focusing on marketplace-specific features
Leverage proven reliability: Benefit from Merpay’s battle-tested payment infrastructure used across Mercari’s marketplaces
Maintain orchestration consistency: Payment processing integrates seamlessly as one step in our Saga workflow

For more technical details about Merpay’s payment solution, you can refer to the blog post by Foghost (Japanese only).

Current Status and Future

We’ve just implemented a basic crossborder transaction flow in this order management system. Going forward, we plan to add more features and roll out to additional regions. We aim to quickly support these expansions by leveraging this foundation.
As the business grows, we can expect to face new challenges. To address these challenges and strengthen the foundation, we will balance infrastructure enhancements with feature development.

Conclusion

In this post, I’ve shared how we designed and built a flexible order management system. The design principles for our order management system—flexible lifecycles and robust orchestration—will be validated through future expansions as we onboard more transaction types and business scenarios, including both crossborder and local transactions.

I’m excited to be part of this project and look forward to sharing more lessons learned as the system evolves.

From Local to Global: Building Seamless B2C Product Integration at Mercari

Thu, 09 Oct 2025 10:42:49 GMT

I am Ahsun, working as a Software Engineer @Cross Border (XB) Engineering. In this article, titled "From Local to Global: Building Seamless B2C Product Integration at Mercari," I’d like to delve a bit deeper into how we architected a robust, scalable product synchronization system that handles both real-time updates and bulk data migrations between Mercari Shops System and Global Foundation. We’ll dive into the key challenges we faced, critical design decisions we made, and learnings that shaped our iterative approach to building a production-ready sync infrastructure.

The Challenge: Connecting Two Product Worlds

At Mercari, we operate in a unique cross-border commerce landscape. Our Japanese B2C marketplace (Mercari Shops) serves many local merchants and customers, while our Global App connects international buyers with Japanese sellers. The challenge? Seamlessly synchronizing millions of products between these two distinct ecosystems in near real time to enrich experience for our customers.

The Business Context

C2C: Single Product for sale.
B2C: Product with multiple variants (e.g. size, color) having distinct stock quantities per variant. So customers can order multiple quantities of each variant.
Mercari Shops System: Japan-focused marketplace with local merchants.
Global Foundation: Cross-border platform serving global customers.
The Gap: Real-time product sync across different data models, currencies, and business rules.

Why This Integration Matters

Key motivations for this integration are as follows;

Enable Japanese merchants to reach global markets effortlessly
Provide consistent product experience across platforms
Maintain data integrity across distributed systems
Support millions of products with sub-second latency requirements

Data Sync – Challenges and Architecture

Here are some challenges and learnings we encountered while building this system, and how we refined our architecture iteratively:

Challenges

Event Deduplication & Ordering: Managing duplicate events and out-of-order message delivery in high-volume PubSub streams required implementing a robust Sync Tracker with message ID-based deduplication and timestamp validation to ensure data consistency.

Dual Sync Strategy Complexity: Coordinating both real-time event-driven sync and batch historical sync through the same ProductSync service while maintaining data integrity and avoiding conflicts between live updates and bulk operations.

Cross-System API Dependencies: Handling API calls to Mercari Shops systems for fetching latest product state introduced latency and failure scenarios that required careful retry logic, rate limiting, and graceful degradation strategies.

Asynchronous Search Indexing: Ensuring search index consistency without blocking the main sync flow by implementing event-driven indexing where ProductInventory publishes events after database storage, allowing SearchIndexer to update indices asynchronously.

Architecture

Our B2C product sync follows a dual-strategy approach, combining the best of real-time and batch processing patterns for old listings. Here’s the high level design for the current architecture.

Key Components

Real-Time Event Processing
- Pub/Sub events from Shop product updates
- Immediate sync for product changes
- Sub-second latency for critical updates
Batch Processing Pipeline
- Handles bulk product imports from BigQuery exports
- Processes millions of products efficiently
- Recovers from failed sync operations
Multi-Tier Service Architecture
- Tier 1 (Admin): Business logic and orchestration
- Tier 2 (Product): Core product management
- Tier 3 (Search): Handling search infrastructure

Development & Release Strategy

Our modular monolith architecture features a database designed to support diverse product types from multiple data sources. With active development across multiple internal modules by numerous contributors, we implemented isolation mechanisms to prevent cross-module interference and maintain shared component stability, so we decided to breakdown our work and scope into three parts,

Handling Events from multiple sources: For shop products we decided to create a separate module that will process all the events and transform them into Global Foundation specific data models. This module only consumes internal product inventory APIs for resources management.

Product Inventory: Created separate APIs for shop products that need special handling considering a product can have multiple variants (e.g. size, color) aspects but it’s developed in a way to reuse the existing internal APIs.

Search & Discovery: We unified the interface to support both C2C and B2C products, implementing the necessary architectural adjustments for compatibility.

Release Mechanism

We divided our data into two categories whose sync approach varies: "Live Data Sync" and "Historical Sync", here I will briefly describe the approaches we took to sync all the data.

Live Data Sync

We handle multiple events (e.g. create/update/delete product, update stock) for active listings with controlled RPS (via our internal PubSub gRPC pusher mechanism) and fetch critical data via APIs for each event to avoid any data stallness.

What is PubSub gRPC Pusher?

PubSub gRPC Pusher provides a subscription type for Google Cloud Pub/Sub that sends messages as a gRPC request. This is an in-house Mercari product not by GCP, designed to achieve high throughput, long running jobs, flexible delivery rates, etc.

To safely import all the shop products into our production environment, we decided to make the following approaches controlled by these configurations.

Steps:

Starts with small, only targeted 1 shop with a small amount of products to verify integrations (e.g. consistency, error handling).
Allow only search indexing but by default search results excluded shop products.
Verify integrations.
Include shop products in the search results via backend feature flags for limited internal users to avoid any negative impacts on our customers’ experience.
Verify end to end integrations.
Whitelisting more shops via configurations to speed up the live data sync.

Historical Sync

To sync old (e.g sold out, inactive) listings or any data anomaly happening after live data sync, we run batches targeting shops incrementally based on minimum products to max products per shop to manage the load in productions.

We use the following configurations for controlling batch processing. By utilizing this configuration we can control multiple aspects of the processing based on our system capacity at different times of the day.

// Sample config.

"admin-b2citemsync": {
job_config: {
        job_id: "JOB_XXXXX"
        start_offset:      "b2c-items/20250925-partition/partition-000-000000000000.json"
        end_offset:        "" // if omit then will target all the files in the partition.
        gcs_folder_path:   "b2c-items/20250925-partition/"
        resource_type:     "MK_JP_B2C_PRODUCTS"
        page_size:         300
        partial_data_size: (100 * 1024 * 1024) // 100MB
        concurrency_count: 500
        rate_limit:        1000
    }
}

Steps:

Run batches in the off-peak hour to avoid unnecessary load in the DB.
Implement phased rollout starting with small-catalog shops, then scale incrementally based on performance validation.
Use the appropriate configurations (e.g. RPS, file size) based on the capacity including dependent services.
Retry partial failed products.
Repeat.

Key Learnings

Summary

Through this comprehensive B2C data synchronization architecture, we successfully solved the critical challenge of reliably syncing millions of products across thousands of shops without compromising system performance or data integrity. By implementing dual synchronization pathways (real-time and batch) with centralized tracking, we achieved zero-downtime rollouts and maintained high-precision data synchronization across all integrated systems. Without this robust infrastructure, we would have faced frequent sync failures, data inconsistencies, and inability to scale beyond small pilot shops—ultimately preventing our cross-border expansion goals and risking significant revenue loss from search index outages.

Detailed Implementation Benefits

Event-Driven Architecture Benefits: Separating concerns through event-driven design (sync → store → publish → index) provided better scalability, fault tolerance, and allowed independent scaling of different system components.

Centralized Sync Control: The Sync Tracker became the heart of the system, providing comprehensive monitoring, deduplication, error handling, and audit trails that were essential for debugging and ensuring data reliability in production.

API-First Data Enrichment: Rather than relying solely on event payloads, fetching complete product data via API calls ensured data completeness and consistency, though it required careful handling of external system dependencies.

Clear System Boundaries: Explicitly defining Global Foundation vs Mercari Shops system boundaries with proper authentication, rate limiting, and error handling made the integration more maintainable and easier to troubleshoot in production environments.

Future Prospects

With this release, we’ve achieved full implementation of the core synchronization infrastructure and foundational data pipeline architecture. Moving forward, our technical roadmap focuses on implementing mission-critical features for cross-border transaction processing, such as product pre-order functionality and authentication features, while rapidly increasing the number of countries we expand to. We need not only horizontal expansion but also localization and growth in specific countries, entering a phase of further utilizing the infrastructure.

Behind the Infrastructure Powering Global Expansion

Wed, 08 Oct 2025 12:00:41 GMT

I’m yanolab, working as an Architect and SRE in Cross Border (XB) Engineering.
On the first day of this blog series, we introduced Rebuilding App and Foundation for Global Expansion. In this article, titled "Behind the Scenes of Infrastructure Supporting Global Expansion," I’d like to delve a bit deeper into the architecture, frameworks, and initiatives of our backend systems.

Background

Mercari has long adopted and operated a Microservice architecture, investing in its ecosystem. We have Microservice templates called echo services, an SDK for developing Microservices in Go, Terraform modules called starter kits that consolidate basic infrastructure configurations, and an SDK that abstracts Kubernetes configurations to manage Deployments with minimal code. Additionally, when releasing Microservices, there’s a process called Production Readiness Check (PRC), and newly developed products or Microservices must pass this checklist. While these ecosystems and processes have matured, the increasingly complex ecosystem has raised the learning cost, and the bloated PRC has meant that launching a single Microservice now takes at least three months. Moreover, when launching new businesses, despite starting with a small team, we often need to launch dozens of Microservices. In such cases, the effort to spend 3 months per Microservice is unrealistic, and Mercari’s recent new businesses have increasingly adopted Monolith-like approaches. (ref: Mercari Hallo’s Tech Stack and Why We Chose It)
In rebuilding infrastructure for global expansion, we anticipate eventually reaching the same scale as the current Mercari Marketplace. Therefore, rather than a simple Monolith, we’ve designed and are operating a special Modular Monolith that maximizes the use of our existing ecosystem while enabling Microservice-like operations.

Modular Monolith with Flexible Deployment

Mercari’s ecosystem, designed for Microservices, is fundamentally based on one repository per service and doesn’t assume large-scale, complex system configurations. For example, our CI/CD assumes one binary, one container, and one Deployment. When deviating from this environment, the implementation side needs to create and maintain custom workflows. To avoid the cost of continuous independent maintenance, the Cross Border team adheres to this policy while enabling Microservice-like operations to distribute operational load as the business grows in the future. The system is compiled into a single binary, but modules can be enabled or disabled through configuration files. Additionally, by defining interfaces between modules with Protocol Buffers and using gRPC for communication, we’ve increased operational flexibility without being constrained to communication within the same instance. This allows us to use the existing CI build system as one binary and one container while enabling Microservice-like operations where modules can be turned on and off through configuration files and communication partners between modules can be arbitrarily configured. Furthermore, by using Protocol Buffers for interfaces between modules, we’ve increased module independence while enabling teams to collaborate on module development from the interface design stage. (Fig. 1)

Fig.1 Modular Monolith with Flexible Deployments

AlloyDB is used as the database for the new infrastructure. In Mercari’s past Monolith, a shared database was used across the entire system, with no restrictions on table joins or permissions across domains. As a result, interdependencies between domains increased as the service grew, and operational costs escalated. In contrast, when migrating to Microservices, Spanner and CloudSQL were adopted by many services and teams. Having each service maintain its own database independently was an excellent choice in terms of domain and service independence, ownership, and maintenance. However, from a cost perspective, it was inefficient for each team to have its own database and maintain an HA configuration for stable operation even with low request volumes, resulting in particularly wasteful costs for services with few requests. Therefore, the Cross Border team decided to use the same cluster as much as possible to save costs, but separate service accounts for each module to restrict accessible databases, and divide databases on a per-module basis. This allows us to keep costs down while preparing for future division and scaling. (Fig. 2)

Fig.2 DB Isolation

Traditionally, Mercari has configured Microservices through environment variables, but with a Monolith, we anticipated that configurations would become extremely numerous and managing configurations across environments would become complex. Therefore, we adopted CUE lang for configuration files, enabling default configurations to be managed from a single source and allowing only values that differ per environment—such as development or production—to be managed as differences. These configuration files are bundled into containers during the container build process, and depending on the environment, the appropriate configuration is automatically used—local configuration for local environments, and corresponding configurations for development or production environments. Additionally, by allowing the standard configuration to be overridden with CUE/YAML at runtime, we’ve also made it possible to apply different configurations for each Deployment. (Fig. 3)

Fig. 3 Difference management of config

For example, we define the standard configurations for development and production environments as the default config as shown below (Fig. 4). In this case, the ProductInventory application in the Product module uses localhost as the address for the Search module.


#GRPCClientConfigSpec: {
    address: string | *"localhost:(#HTTPPort)"
    timeout: =~"^([0-9]+(\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$" | int | *"3s"
    retry:   int & >=0 | *3
}

components:
    "layers/tire2/product/applications/productinventory":
        enabled: bool | *false
        search_module: #GRPCClientConfigSpec
    "layers/tire3/search/applications/productsearch":
        enabled: bool | *false
    ...

Fig. 4 Common part of development and production

Suppose we define the common configuration for the development environment as shown below (Fig. 5). In this case, all features are enabled both in the GKE environment, which is part of the development environment, and in the local environment, where all modules use the modules on localhost.


components:
    "layers/tire2/product/application/productinventory":
        enable: true
    "layers/tire3/search/applications/productsearch":
        enabled: true
    ...

Fig. 5 Development specific configuration（Enabled all of modules）

When separating GKE Deployments in the production environment, we mount a ConfigMap as YAML separately from what’s bundled in the container and load it. For example, by setting the connection destination of the Inventory application in the Product module of DeploymentA to DeploymentB (Fig. 6), and enabling only the ProductSearch application of the Search module in DeploymentB (Fig. 7), it becomes possible to operate only the Search module independently.


components:
    "layers/tire2/product/applications/productinventory":
        enable: true
        search_module:
            address: deploymentB.xxxx.svc.local
    "layers/tire3/search/applications/productsearch":
        enable: false
    ...

Fig. 6 The Search module used by the Product module can be switched to a different Deployment


components:
    "layers/tire2/product/applications/productinventory":
        enable: false
    "layers/tire3/search/applications/productsearch":
        enable: true
    ...

Fig. 7 Deployment with only the Search module enabled

This flexible architecture enables operation as a single binary in local development and development environments, while allowing modules to be appropriately separated and operated in the production environment. This is particularly powerful for local development, eliminating the challenge of Microservice development where you need to prepare an execution environment including dependent Microservices, thus dramatically improving the efficiency of development environment setup and maintenance. However, in this infrastructure rebuild, we’re not replacing all Microservices, and dependencies on existing Mercari Microservices still exist. To handle these dependencies, we use a product called mirrord to connect from the local environment to the remote Kubernetes environment for development. We also use a product called air, which enables dynamic reloading of changes, achieving a modern development environment similar to web application development.

Adapting to Change with a Monorepo

In Mercari’s Microservices, we create a repository for each service and operate the Protocol Buffer definitions, infrastructure management using Terraform, and Kubernetes deployment environment repositories as monorepos shared by everyone. While this approach is effective, being different from the main repository requires moving between repositories. The frequent occurrence of this context switching is extremely stressful for developers. Additionally, automation across repositories not only takes longer to process due to individually running CIs, but when issues occur, it’s difficult to understand where and what is happening, which worsens the developer experience. In this infrastructure rebuild, to review these developer experiences, we’ve also reconsidered this structure and are attempting to consolidate the Backend project, Frontend project, Protobuf definitions, and Terraform in one place so that development can be completed within a monorepo as much as possible. (Only Kubernetes deployment uses the existing monorepo due to ecosystem constraints.)

By clearly defining boundaries with Modular Monolith while managing not only Backend projects but also Frontend projects in a monorepo, we’re making it easier to contribute across languages and roles while aligning applications, architecture, and frameworks. In terms of maintenance as well, we believe efficiency is high since we only need to maintain one location for scripts, workflows, CI, etc. At Mercari, we had long been unable to visualize organizational and team productivity, and accurately measuring developer productivity was a challenge. Since 2024, we’ve introduced DX with the aim of visualizing and improving developer productivity. DX combines qualitative data from surveys with quantitative data such as productivity-related metrics from GitHub to visualize four aspects: efficiency, speed, quality, and novelty. We found that the monorepo approach produced better results in these values compared to Mercari’s overall scores.

One slightly unique aspect of the monorepo we built is that we use Terraform and CUE lang for infrastructure management (the traditional tf format is also available). In CI, we convert from CUE to JSON and apply it. By defining infrastructure in CUE, environment construction with difference awareness becomes possible, similar to the configuration management of the Modular Monolith introduced above. Since CUE can be merged and used with YAML and JSON, we feel it’s extremely effective for automation. Going forward, we have the ambition to leverage the advantage of having all monorepo data in the same repository and work on Framework defined Infrastructure that automatically generates infrastructure configuration files from Modular Monolith configurations and frameworks. (Fig. 8)

Fig. 8 Framework Defined Infrastructure

Approach to Increasingly Complex Domains and Dependencies

Currently, Mercari has several hundreds of Microservices related to the Marketplace business as well as Merpay. These services are not only divided more finely than necessary and interdependent with each other, making maintenance difficult, but they also make it extremely challenging to determine which Microservice should receive new functionality, which Microservice’s features can be utilized, or whether a new Microservice should be created in the first place when trying to create new features. Therefore, as Cross Border rebuilds the Marketplace infrastructure from scratch, we’ve been proceeding while organizing domains and roles by introducing the concept of Tiers and dependency maps, focusing on specific functions like the Like service, and re-consolidating services that were divided too small—such as bringing them together into a Social module—into reasonably large domains.

In this Tier concept, we’ve divided roles into five layers—BFF (Backend for Frontend)/Gateway, Tier 1, Tier 2, …Tier 4—and added roles and restrictions for each layer.

BFF/Gateway Layer

BFF is well known, but this layer defines APIs optimized for Mobile and Web screens, and all requests are sent through the BFF before being passed to lower layers. Language and currency conversion based on customers is also handled by this layer. It is jointly owned and maintained by Mobile engineers, Web engineers, and Backend engineers.

Tier 1

Primarily responsible for request orchestration and business flows. The responsibility of Tier 1 is to build business processes using modules in Tier 2 and below. The image is that it builds processes using various Marketplace features, so it’s the area responsible for horizontal processing.

Tier 2

Primarily a domain-specific layer that realizes Marketplace’s core functions. This includes modules like Product and Order. The image is that it’s the area responsible for vertical processing specific to the relevant domain.

Tier 3

This layer basically provides more generic functions that don’t depend on the Marketplace. This includes Search and Notification.

Tier 4

This layer is somewhat special and provides modules that must meet specific requirements or functions that are difficult to belong to Tiers 1-3. We place modules that exclusively handle personal information with different security and operational requirements from other modules in this layer.

We’ve imposed the constraint that requests always flow from top to bottom and communication between modules in the same Tier is prohibited. However, we’ve established a rule that when accessing from an upper Tier to a lower Tier, intermediate Tiers can be skipped, and access from BFF to Notification is permitted. (Fig. 9) Databases are also separated by module, and it’s not possible to span transactions across modules. These rules greatly increase module independence while preventing the proliferation of small modules. If communication between modules in the same Tier becomes necessary, it indicates that the domains of those modules are very similar, and we view it as a good signal to review domain boundaries.

Fig. 9 Tier Concept

The infrastructure rebuild has only just begun, but by utilizing well-defined and stable service groups such as Payment and IdP while reorganizing and implementing Marketplace domains using this design methodology, we’ve been able to keep it to 18 modules as of October 2025.

Current Challenges

Currently, to enable deployment on a per-module basis, we manage versions per module in files and detect version upgrades for each module by incrementing those versions at release time. However, this method is incompatible with GitHub Flow, which uses the main branch for releases, and there’s a risk of unintended changes being included in releases. We’re currently working through trial and error to solve this problem.

Future Developments

In these times when AI-driven development is becoming mainstream, quickly launching new businesses is necessary to secure competitive advantage. The Cross Border team’s Monorepo and Modular Monolith approach introduced here has a reasonably high initial construction cost, so we’re working with the Platform team to make it easier and faster to build so it can be applied to Mercari’s future new businesses. If there’s an opportunity somewhere down the line, I’d like to write another article about these results.

Finally

On November 13, 2025, the Mercari Group tech conference "mercari GEARS 2025" will be held.

Please join us! Registration is here 👉 https://gears.mercari.com/

Tomorrow’s article is by @Gary. Please continue to enjoy "Series: Behind the Scenes of Developing ‘Mercari Global App,’ Mercari’s First Universal App."

Rebuilding App and Foundation for Global Expansion

Tue, 07 Oct 2025 10:04:35 GMT

This is @deeeeet from Cross Border (XB) Engineering.

As we shared at our recent business strategy presentation, we have released a new global version of the Mercari app to further accelerate Mercari’s global expansion.

This app is a new application, different from the currently available Japanese and US versions of Mercari, and we have also rebuilt the backend infrastructure from scratch. In this article, I’ll introduce the strategy and architecture of the global app and its infrastructure from an engineering perspective, while reflecting on the lessons learned from Mercari’s past challenges.

Cross-Border Transactions at Mercari

Some of you who have listed items on Mercari Japan may have experienced your products being "proxy purchased" by businesses rather than general customers. This is made possible through a cross-border (XB) transaction system that allows overseas customers to purchase items listed on Japan’s "Mercari."

Cross-border transactions at Mercari are realized through partnerships with proxy purchase partners. Overseas customers first order Mercari products on partner websites. The partner then purchases the items on Mercari as a proxy buyer and handles the payment process. Domestic sellers ship products to the partner’s designated warehouse in Japan, just like regular domestic transactions. After the products arrive at the warehouse, the partner inspects and repackages them for international shipping, then sends them to overseas customers.

This system benefits both overseas and domestic customers. Overseas customers can easily purchase unique Japanese products without worrying about language barriers or currency differences. Meanwhile, domestic customers can expand their sales opportunities globally without any need for direct communication with overseas customers or complex international shipping procedures – they can sell just like in domestic transactions.

This cross-border transaction business started in 2019 and has grown significantly in recent years, with GMV growing 15x over the past three years. Anime, comics, games, and entertainment-related goods categories account for much of the total transactions, showing strong demand from overseas customers.

Given this strong demand and growth, in addition to the proxy purchase partner site system, we also started an initiative to enable proxy purchases through Japan’s Mercari web service. This system allows overseas customers to create accounts directly on Mercari and search for and purchase products through the Mercari experience (while still maintaining the partner company intermediary transaction). We released this initiative in 2024, and it’s currently available in Taiwan and Hong Kong, with growing user numbers.

While this cross-border transaction business has grown steadily, several important challenges have emerged. As explained below, the existing JP system was built specifically for the Japanese market and designed with single-currency and single-language assumptions. Since cross-border transaction features were added on top of this, there were limitations to expanding to multiple countries and adapting to each country’s unique business practices. There was also a competitiveness issue with only web version availability, especially in Asian markets, where most EC usage is mobile-based.

Despite these challenges, demand from overseas markets clearly exists, with particularly high interest in anime and game-related products. While currently limited to Taiwan and Hong Kong, similar potential demand clearly exists in the US and EU markets. To maximize this opportunity, we needed a new approach to expand to more countries faster.

Therefore, we decided not to simply extend the existing system, but to build a new application and infrastructure designed for global expansion from the ground up. This was a strategic decision looking ahead from cross-border transactions to eventually launching local marketplaces in various countries and ultimately realizing a global marketplace that transcends borders.

Approach to Global Expansion

Realizing a global marketplace has been Mercari’s vision since its founding, and this is not our first challenge in global expansion. We have challenged ourselves with business expansion in the US and continue to focus on its growth. We also have experience attempting expansion into the UK in the past.

In previous global expansions, we took the approach of building local C2C marketplaces from scratch in each country, similar to Japan. However, the latest global expansion takes a new approach, learned from the successes and challenges of cross-border transactions. We’re adopting a strategy that focuses on "cross-border transactions," delivering products from Japan to overseas as the business axis, then gradually expanding services while leveraging the customer base built there. The expansion pace is also significantly different from before, aiming for 50 countries within 3 years. This represents a shift in strategy to start by delivering the unique and abundant products listed by Japanese customers and businesses to the world, then exploring further possibilities from there.

This shift in business strategy has also significantly changed our engineering strategy.

Previous expansions in Japan, the US, and the UK were each realized through independent, different systems. Of course, initially, we took an approach of deploying a common codebase to each country (though with separate data). However, due to code complexity from adapting a system built for Japan to each country’s circumstances (e.g., “if” statements for country switches written in many places) and decreased decision-making speed in each country due to the need for alignment between countries, we ultimately decided to fork, resulting in independent systems with separated development and operation structures for each. The US subsequently redesigned its app to match local UI/UX and implemented unique features on top of it, so Japan and the US systems remain separated today.

This method was effective for quickly launching businesses and deeply optimizing for each country’s market. Creating independent organizations and developing systems for each country’s business growth was also important. However, from a longer-term perspective, the following challenges made it difficult to connect to the next expansion:

Cost and speed of expansion: From the perspective of increasing the number of countries, common infrastructure wasn’t prepared, and when considering the next country, we would need to rebuild new applications and backend infrastructure.
Inefficiency of development resources: Similar features were implemented individually in each country, requiring dedicated teams for each infrastructure, causing duplication and inefficiency of development resources.

"Cross-border transactions" themselves are alreayd built on the existing JP system. However, as described in more detail below, the existing system has become complex, and there were limits to expanding countries faster and providing better UI/UX for global markets. And connecting to what comes after "cross-border transactions," such as launching local marketplaces in new countries, is extremely difficult.

To fundamentally solve these challenges and efficiently accelerate new international expansion centered on "cross-border transactions," we needed a new strategy. So we established a new vision of "supporting all countries and regions with a single global infrastructure rather than building individual systems for each country or region" and began developing that infrastructure.

Global Foundation Development Strategy

Several approaches were considered for developing this single global infrastructure, and we chose a "hybrid approach of extension and reconstruction." Let me explain the background leading to this approach through the evolution of Mercari’s backend systems.

Evolution of Mercari’s Backend Systems

Mercari’s backend system started as a Monolith architecture (implementing all features in a single codebase). This is why we could choose the fork option when starting US and UK businesses (though duplicating the many mechanisms and tools supporting each country’s scale in the infrastructure behind the scenes wouldn’t have been easy).

Around 2017, the scale of the Japan organization began to expand rapidly. Organizational growth made it difficult for many people to develop simultaneously in a single massive codebase, and bugs in some features often caused failures that affected the entire service. Additionally, most systems were built on-premises, and their operation and expansion became bottlenecks. To solve these problems, we began migrating to Microservices architecture and the cloud (along with transitioning to DevOps). I joined just before this, and have been responsible for promoting the migration project and establishing and expanding the Platform Engineering team that prepares the foundation and tools for Microservices development.

We adopted the Strangler pattern as our approach to Microservices architecture migration. This involves placing a Gateway in front of the existing system and gradually migrating traffic to the new system around that Gateway. More specifically, we repeatedly (1) extract feature groups implemented in the existing system as Microservices and (2) route usage traffic for those features from the Gateway to the Microservices side, gradually migrating to the new system. Several years have passed since migration began, and we’ve extracted many features from the Monolith and developed new features on top of them. Cloud migration for almost all services is also complete (over 100 services).

After Microservices migration, Japan began launching multiple new businesses in addition to the main C2C marketplace business. These include Merpay for fintech, Mercoin for cryptocurrency, Mercari Shops for B2C business, and Mercari Hallo for on-demand work. Merpay extracted Mercari’s payment system and built it as Microservices architecture on the same infrastructure platform as C2C. Mercoin has largely separated infrastructure for security but basically develops with similar architecture patterns. Shops has Microservices architecture but is an independent system separated from C2C (while it’s one mobile app, the backend is separated).

Alongside these years of Microservices migration and multiple business launches, we have also promoted the development of common infrastructure. Not only development infrastructure and tools at the Platform engineering layer that I’ve led, but also foundation that can be used across multiple businesses like ID platform, payment platform, and marketing platform.

This is the evolution of Mercari’s backend systems since its founding.

Challenges with Existing Systems

Looking at the existing systems holistically in 2025, there are several challenges, but the biggest is that core functions important for the C2C marketplace remain in the Monolith infrastructure. While we’ve been able to extract some features as Microservices using the Strangler pattern, this approach only extracted upper-layer features as proxies and didn’t progress to data migration in many areas (meaning dependencies for data retrieval remained). In particular, we haven’t been able to extract very important C2C functions like "transaction management" and "shipping" from the Monolith and its DB. A major reason is that these two have strong logical coupling that couldn’t be separated easily. Therefore, strong dependencies on the Monolith still remain. While this area still requires much development and changes, it remains on a complex codebase, requiring urgent action. As someone involved in Microservices migration from the beginning, not tackling these important parts early is a major regret.

Looking at global expansion, this becomes a major challenge. Transaction management and shipping systems remaining in the Monolith are designed specifically for the Japanese market. Transaction management assumes only Japanese yen, and adding support for multi-currency transactions, exchange processing, and different tax systems in each country would be very costly. The shipping system is also tightly coupled with Japanese domestic carrier systems, making it difficult to support local carriers in each country and different shipping options without fundamental rebuilding.

There’s also the system divergence problem between C2C marketplace and B2C Shops. Currently they have separate transaction and shipping systems, and product management is also separated, resulting in inability to provide a unified experience even to the Japanese customers. This is due to independent services being considered in the original vision, and even when the direction changed to integrate them, execution was difficult due to the Monolith problem above.

There are also challenges with Microservices architecture itself. As a result of emphasizing ownership and freedom for each service and not achieving sufficient abstraction between services, and by not properly separating domains and making division units very small, many small Microservices with slightly different implementations were built. This has made Microservices operation costs very high. Mercari frequently reorganizes to move forward with speed, but each time requires transferring Microservices ownership, and implementation differences increase onboarding costs.

Due to these constraints, it became clear that proceeding with global expansion as an extension of the existing system had both technical and business limitations.

Direction for Global Foundation

Based on this evolution and current challenges, we considered several approaches for developing the global foundation. First, taking the fork option, like past US expansion, has become very difficult. Duplicating many microservice systems is not realistic. We also considered rebuilding everything from scratch, but excluded this option from a cost and efficiency perspective. In conclusion, we chose a "hybrid approach of extension and reconstruction of existing systems."

In this approach, determining where to draw the line between extension and reconstruction was important. Many existing systems are specialized for the Japanese market, and many services have been converted to Microservices. Extending all of them wasn’t realistic, and since the Japan business continues to be important, it was also important that global expansion could proceed independently from it. We also had a strong desire to avoid global dependencies on the remaining Monolith.

For "extension," we mainly decided to utilize the common infrastructure developed alongside multiple business launches. We specifically selected services that require strong expertise and are designed with extensibility in mind. As described in detail below, we’re also considering moving away from Microservices, deciding to depend not on small, detailed services but on sufficiently large and independent "domains" (at a level that could be replaced by SaaS). Based on these criteria, for example, we’re making the ID infrastructure globally common, and connecting the payment infrastructure to Stripe through the Merpay infrastructure to support global currencies and local payment methods. We’re also utilizing existing systems by extending search infrastructure, marketing infrastructure, and others.

Other parts take the "reconstruction" option. In particular, the aforementioned C2C service "transaction management," "shipping," and "item/product management" had to be rebuilt. To avoid the same problems as Japan, we’re building with consideration for (1) making each loosely coupled for easier long-term extensibility, (2) treating C and B products equally to provide unified UI/UX, and to enable multi-country expansion and new local marketplaces in other countries, (3) flexibly supporting each country’s currency, language, tax/customs systems, and regulations (assuming “Design for Two” – see Tenets below), (4) being able to handle products and shipping methods from countries other than Japan.

Also, simply rebuilding would just create a new, separate infrastructure. While initially focusing on global success, we’re moving with the assumption of eventually replacing Japan’s C2C and B2C infrastructure as well (having actually achieved release, we’ve started a project to utilize this infrastructure in Japan too).

For mobile apps and Web, a different UI/UX is essential globally, so we chose to rebuild. Additionally, by renovating the backend, we can switch the API itself and improve implementation.

From Microservices to Modular Monolith

To tackle the challenges of Microservices architecture described above, we’re developing the "reconstructed" backend infrastructure as Modular monolith architecture.

Challenges of Microservices

The main reason Microservices architecture operation costs became high at Mercari is that we gave too much development freedom to each service. We’ve promoted minimal technology stack unification: Go for server implementation, Spanner/CloudSQL (MySQL) for databases, and Kubernetes for infrastructure. On the other hand, the repository strategy was Polyrepo (1 service = 1 GitHub repository), and while there were baseline templates and minimal common libraries, repository structure and implementation policies were left to each team. Therefore, while they’re all Go Microservices at a macro level, quite different services were mass-produced at a micro level. Even if each service’s operation cost is small, when you need to manage multiple different services, the differences prevent standardization, increasing costs.

Additionally, Mercari moves forward with speed and frequently changes direction, so organizational changes are frequent. This requires frequent Microservices ownership transfers. Each transfer requires onboarding, and implementation differences increase that cost. It also makes promoting standardization difficult.

Also, especially on the C2C side that migrated from Monolith, there are many areas where proper domain separation wasn’t achieved, with low service cohesion in many places. This requires changes across multiple services and teams for feature additions, leading to increased communication costs. Strengthening ownership for each service conversely, made it harder to accept changes from outside.

The approach that successfully addressed these challenges with Microservices architecture implementation was Mercari Shops’ Monorepo approach. This method puts all Shops-related Microservices in one repo, achieving abstraction and unification of implementation between services, reducing operation costs from multiple services. It provides a Monolith-like development experience while services are separated and deployed behind the scenes (gaining fault tolerance benefits), incorporating the best of both worlds.

However, even this approach has challenges. Managing and maintaining infrastructure and automation mechanisms for this Monorepo is very costly (being built largely separate from existing Platform prevented collaboration with the common infrastructure teams). Testing, deployment, and development environment construction for Microservices inevitably becomes complex. For example, the test environment takes a wealthy approach of duplicating all services for each PR. They also strictly separate DBs for each service, increasing infrastructure costs.

Modular Monolith

Given this background, we chose Modular monolith architecture for building the new infrastructure. It’s not just Modular monolith but designed to deploy specific Modules independently when necessary (close to the Service Weaver concept).

I believe the initial Mercari Monolith was unable to properly separate domains and modules, causing code tight coupling and resulting complexity. We’re avoiding similar problems by clearly organizing service boundaries and dependencies for each module. We’re avoiding complexity from over-separation like Microservices, creating modules with sufficiently condensed functionality. At the same time, we’re also enabling Microservices-like fault tolerance benefits by allowing independent deployment when necessary.

In the initial development phase with not many people, we basically don’t limit ownership to specific modules (though of course some people are stronger in specific areas). We want everyone to have ownership of the entire codebase. This allows module assignments to be dynamically determined by product development priorities, eliminating the wasteful communication coordination costs that occurred with Microservices. Meanwhile, even as the organization expands in the future, assignment by module unit is possible, and there’s room to solve the problems we encountered with the previous Monolith.

Being a Monolith makes local development environment construction easy with one binary, and testing and deployment become simple, eliminating development burden caused by Microservices and creating a better development experience. Infrastructure and CI/CD infrastructure can directly use what the Platform Engineering team provides, avoiding the infrastructure operation costs that the Shops Monorepo approach fell into.

However, this policy is new within the overall organization, and there’s the challenge of how to coexist with the existing Microservices approach. Realistically, it’s not easy to return all separated Microservices to a Monolith. Therefore, Microservices themselves will likely remain in the future. To reduce Microservices development and operation costs, it’s important to adjust service division units to more appropriate levels, and furthermore, increase unification through Monorepo approaches like Shops achieved. And for future new businesses, unless there are special reasons, I think we shouldn’t choose Microservices architecture as the first move. We’re also considering horizontally expanding this global infrastructure’s Modular monolith construction pattern and standardizing implementation patterns.

Technology Stack

Below is a list of technology stacks used in building this infrastructure. Basically, we’re not making major changes from the stack Mercari has cultivated, but utilizing them well.

Infrastructure: We continue to use Google Cloud as our main cloud. The main region is Tokyo, but we’re considering using other regions in the future (especially from a performance perspective). For application execution infrastructure, we use Kubernetes (GKE) managed by the Platform Engineering team.
Database: We chose AlloyDB for the database. While we’ve mainly selected Spanner centered on Merpay, we chose AlloyDB considering (1) avoiding lock-in as much as possible, considering we might not be able to handle everything with Google Cloud when considering long-term expansion, and (2) using the better development experience ecosystem with PostgreSQL. We’re also considering CockroachDB and may consider switching depending on future expansion.
Languages/Frameworks: Go for servers, Swift for iOS, Kotlin for Android, and Next.js (TypeScript) for Web. We haven’t changed much here.
Monorepo: More detailed blogs will be written later, but iOS, Android, and Web are each developed by extending JP service repositories as Monorepos. By extracting modules that can be shared between JP and global and unifying CI/CD, we’re improving development and operation efficiency.

Tenets

This backend infrastructure development includes many members from our India base as well as Japan. For members from various backgrounds to realize the direction introduced above, it’s important that everyone can make decisions following the same guidelines. To achieve this, we established "Global Engineering Tenets." Tenets are inspired by Amazon’s Tenets: supercharging decision-making.

Let me introduce some main Tenets:

Design for two: In software development, you probably intuitively understand that it’s easier to increase support for a feature from 2 to 3 than from 1 to 2. For example, if an application already supports two languages, adding a third language is easy. On the other hand, if an application only supports one language, adding a second language requires much preparation like i18n mechanisms. The same applies to global expansion. Adding new regions or countries to infrastructure already supporting multiple countries/regions is much easier than extending an application for a single region/country. We always assume 2 or more countries in feature and system design.
Global by default but enable localization: While advancing system development for global use, we don’t just expand business to multiple countries but implement localization measures in major markets. Therefore, systems need to be quickly and easily expandable to multiple countries while also having flexibility to support specific country requirements. In the long term, we may establish local engineering teams for localization, and they need to be able to independently develop localized features.
Learn and unlearn from past experience: We have many parts that are newly "reconstructed" this time. However, this shouldn’t be completely new but should utilize past learnings introduced above as important assets. I’ve explained the overview, but there are challenges to review in various areas like mobile development, web development, product development, etc. We strongly asked newly hired members to utilize these as well.
Keep each country’s business isolated: Even when utilizing existing infrastructure and platforms, they shouldn’t affect each other. For example, we need to avoid bugs or incidents occurring globally affecting JP business, or vice versa.

Future Work

With this release, we’ve completed implementing basic functionality. Going forward, we aim to implement features important for cross-border transactions, such as B product pre-order functionality and authentication features, while rapidly increasing the number of countries we expand to. We need not only horizontal expansion but also localization and growth in specific countries, entering a phase of further utilizing the infrastructure. Also, as introduced above, the infrastructure itself is designed to be usable in JP, and we’ve started that replacement project.

We Asked PyCon JP 2025 Attendees What Percentage of Their Code is AI-Generated

Mon, 06 Oct 2025 15:28:28 GMT

AI is rapidly advancing, and AI-assisted coding is becoming integral to software development. As an ML engineer, I’m fascinated by AI’s role in coding, and I wanted the Python community’s take on two questions:

How long have they been using Python, and what percentage of their code is AI-generated now?
What are their top sources for staying up to date on AI/ML news?

PyCon JP is the largest Python conference in Japan. PyCon JP 2025 was held in Hiroshima from September 26-28, and Mercari was a gold sponsor. I, Prashant, along with Yasuhiro Shiwaku and Tomoko Suzuki from the Engineering Office, led Mercari’s sponsorship effort. With help from my teammates (@ayato, @bosco, @kanta, @wakuchan), we ran Mercari’s sponsor booth and talked to attendees about these two questions.

Let’s get to what people said!

AI Generated Code Percentage

Over the last couple of years, I’ve experimented a lot with generating working code using AI. From copying and pasting code between the ChatGPT/Claude web interfaces and my code editor to using agentic coding tools like Claude Code, Cursor, Cline, Codex, and GitHub Copilot, I have tried them all. In the last 6 months, I estimate Claude Code has written about 80% of my code. This is a drastic change in the way I write code now compared to when I first started writing code.

PyCon draws people with a wide range of Python experience, from people who started last year to people who have been using Python for years and years. Given my experience with AI-generated code, I wanted to see how AI has affected the workflows of people with different levels of Python experience.

40 attendees shared their years of Python experience and the percentage of their code that is AI-generated. I have visualized their responses in the bubble chart below.

Responses spanned 1 to 15 years of Python experience, and the median developer reported 50% AI-generated code. Using Python and pandas, I analyzed the data and found a few more insights:

AI adoption does not correlate with years of experience. The Spearman correlation was near zero (-0.037).
Adoption of AI is high across the sample. A majority of developers (62.5%) generate at least half their code with AI, and 27.5% of developers (11 out of 40) generate 80%+ of their code with AI.
After bucketing experience into 1-3, 4-7, and 8+ years, each group had similar average and median AI-generated code percentages, both near 50%. For the 4-7 years group, the median was slightly higher at 60%.

AI/ML News Sources

News about major updates to open-weight and proprietary LLM performance, benchmarks, agentic coding tools, research advancements, and image generation has become common. Almost every week, we see one or more major updates in these areas, and a myriad of minor ones. There is also a whole lot of knowledge shared by the community about best practices for AI tools and how people get these tools to work best for their use cases.

With this rapid pace of development and a constant stream of updates, months feel like decades, and what we learn becomes obsolete in a couple of months. Personally, I find staying up to date quite challenging. Now, you might say keeping up with every new thing isn’t necessary, and that there are always shiny new things in tech to chase. But I’d argue staying on the sidelines is also not an option. Whether we like it or not, software development has fundamentally changed in the last couple of years and will continue to do so. I don’t want to end up being the old man yelling at the cloud. With more and more companies around the world already mandating the use of AI in development and incorporating it into performance reviews, it’s in our best interests to understand the tools we need to use. So I asked the Python community which sources they use to keep up with this rapidly advancing field.

44 people shared how they stay up to date with AI/ML news. 31 listed a single source, while 13 listed two or more. I have compiled the sources in the bar chart below.

X/Twitter was the clear winner, mentioned by 45.5% of respondents (20 out of 44). Learning from co-workers and Zenn were a distant second, each mentioned by 18.2% of respondents (8 out of 44). I didn’t expect "talking to coworkers" to rank highly, but it makes sense. YouTube was third with 15.9% (7 out of 44). Also, I was surprised to see so few mentions of Hacker News.

Conclusion

Mercari Sponsor Booth at PyCon JP 2025

Survey Responses by PyCon JP 2025 Attendees

Our informal survey at PyCon JP 2025 offers a snapshot of AI’s impact on the Python community. The data suggests AI-assisted coding is now a standard practice, with a median of 50% AI-generated code reported across all experience levels. The fact that both newcomers and veterans report similar adoption rates suggests we’re witnessing a fundamental shift in how code gets written, not just a trend among early adopters. This widespread adoption is supported by a fast-moving information loop, where developers rely on X/Twitter and direct collaboration with coworkers to keep pace.

While the sample size for the survey is small, it indicates a significant shift in developer workflows. Thanks to everyone who shared their experiences; I’m keen to see how these trends evolve over the coming years.

Behind the Scenes of Developing Mercari’s First Global App, “Mercari Global App”

Mon, 06 Oct 2025 09:27:58 GMT

Hello. I’m @deeeeet from Cross Border (XB) Engineering.

On September 30, 2025, we announced a new strategy for our cross-border business and launched Mercari’s first globally unified app, the “Mercari Global App” (hereinafter referred to as the Global App).
This time, we’re launching a new series that takes you behind the scenes of our global app development projects.
Stay tuned for topics spanning not just backend development, but also mobile development, web development, SRE & Enabling, and much more.

Global App overview　

The Mercari Global App will enable overseas buyers to browse and purchase items from Mercari and Mercari Shops in Japan. The Global App solves problems related to language, payment, and complicated procedures, providing overseas buyers with an easy, safe, and secure shopping experience similar to that of Mercari in Japan.
The Global App will be available in Taiwan and Hong Kong starting from September 30th, 2025, and is planned to gradually expand to more countries and regions in the future.

Publishing schedule

Following is a collection of links to each article. I recommend bookmarking this page for the prompt update, and it will be very useful if you want to check it out at a later date.

Title	Author
グローバル展開にむけたアプリと基盤の再構築	@deeeet
グローバル展開を支える基盤の裏側	@yanolab
From Local to Global: Building Seamless B2C Product Integration at Mercari	@ahsun
Order management in Mercari Global Marketplace	@takady
The Journey of User-Generated Content Translation	@aymeric
グローバルWebを支えるSREの裏側 — 開発を加速させるための改善アプローチ	@hatappi
Toward a Global Identity Platform	@gia
開発者全員が書けるE2Eテスト ─ 普通のgo testで実現するテスト基盤	@ryotarai
グローバルなメルカリの検索バックエンド設計と検索基盤拡充	@shinpei
Building a region‑aware, SEO‑friendly global web app	@gary
モジュラモノリスの品質を支えるリーダビリティチーム ― AI時代のスケーラブルなコード管理	@osari.k
How We Deliver Mobile App Updates Faster	@manoj
Evolving Mercari’s iOS codebase into a multi-product monorepo	@shingt
Enabling internationalization in our web Turbo monorepo	@gary
The AI Lied to Me — And That’s When I Learned How to Use It	@andrei
Taming Agents in the Mercari Web Monorepo	@maxi
BenchMarking Databases For Global APP	@amit
Behind the Global Launch: Decoding the Android Engineering Strategy for Our New App	@Karthi
Data-fetching strategy for Mercari Global Marketplace Web App	@vb
TBD: How we overcome Project management challenges (How to plan a product launch in 6 months)	@g-bansal
Guest post from FT payment platform — Engineering for Multi-Currency and Multi-Provider Payments	@ryuyama
TBD	@manas
TBD: distributed transactions on checkout flow, specially error handling, retry	@ahsun
Something about global payment and checkout	@huhu
TBD: Ops development with AI	@waiting.lau
Sync Saga	@Shishir
TBD: High output teams	@Atif
TBD: Ordering Features	@Shreyasi
TBD	@Chong (チョン)
TBD	@chris

If any of these articles catch your interest, bookmark this page or follow and check out the official X account for engineers!

Locked Shields 2025 Event Report

Tue, 29 Jul 2025 10:00:21 GMT

Introduction

Locked Shields 2025, the world’s largest cyber defense exercise, was held in early May by the NATO Cooperative Cyber Defence Centre of Excellence (CCDCOE). In the 2025 edition of this event, about 4,000 people from approximately 40 countries formed 17 multinational blue teams to participate in a scenario where they had to defend ICT infrastructure equivalent to that on a national scale.
Similar to last year, three members of Mercari’s Security Team participated in Locked Shields this year. In this article, we’ll share the knowledge we gained on the front lines of international joint exercise.

Team introduction

Three Mercari employees participated in this exercise.

Yuto Iso: Mainly in charge of preserving all information systems under Japan’s sphere of defense and preventing breaches of essential systems.
Hiroki Akamatsu: In charge of vulnerability hunting and fixing for platforms and web applications.
Sana Okumura: In charge of analysis of signs of breaches, confirmation of evidence, and reporting.

The three of us detected signs of attacks, checked evidence of attacks, and identified and addressed vulnerabilities.

Task details

In Locked Shields, participants must defend a large number of information systems from sophisticated cyber attacks. Yuto developed a mechanism to automatically examine all target information systems, which significantly reduced the effort needed to safeguard and restore the systems. This also contributed to identifying vulnerabilities before they were exploited and swiftly recovering systems after attacks.

The Locked Shields scenario contains various services, authentication infrastructures, and networks, including AI features, as well as a platform on which all of those operate. As someone with knowledge of AI, web applications, and container technology, Hiroki supported the team in aspects such as making multiple web applications more robust and building a safe container deployment environment.

Throughout the scenario, attackers try to breach information systems using various attack patterns. Sana checked various forms of evidence to accurately identify and report the extent of impact of attacks, which contributed to the detection and containment of attacks.

Takeaways and results

Each of us approached the exercise from our areas of expertise, but throughout the event, we encountered attacks in areas we had no experience in, such as operational technology systems, so we learned a lot as we worked to defend the systems from countless attacks.

On the less technical side, the event also provided us with hands-on experience in how to smoothly communicate and collaborate with participants specializing in other fields in a cybersecurity defense scenario.

Conclusion

Locked Shields is the only exercise of its kind to such a large scale, and this year’s event was a very valuable experience for the Mercari members involved. We each demonstrated our expertise to our fullest potential to provide wide-reaching technical support to the systems we were in charge of. Through automating system examination and preservation, and rapidly addressing vulnerabilities and identifying extent of impact in a complex environment, we were able to polish our practical skills. In addition, we also gained knowledge of new attack methods and defense strategies.

We were especially reminded of the importance of communication and collaboration in cyber defense through our cooperation with specialists in various areas from other countries. We felt first-hand how difficult it is to swiftly and accurately share information and work toward a shared goal in a constantly evolving situation, and how big the sense of accomplishment is when you overcome that difficulty.

We’re confident that the knowledge and experience gained through this exercise will contribute significantly to strengthening the security of Mercari’s services, enhancing our incident response capabilities, and preparing for potential future cyber attacks. Going forward, Mercari will continue to strive to actively participate in international initiatives such as Locked Shields in order to enhance our cybersecurity technology and provide safer and more secure services.

Source: https://x.com/ModJapan_en/status/1920769117180641309?t=9Bi-vyX3tawRB5F_QAJidw&s=19

How QAs should see AI – A Report from the QA Conference

Tue, 22 Jul 2025 11:11:55 GMT

Hello. I’m @uni0110 from the Merpay QA team.

In June, I attended the EuroSTAR Conference in Edinburgh, Scotland. EuroSTAR is one of the most famous QA conferences in the world. This year, over 60 tutorials, sessions, and keynotes were held over four days. It was a large conference with more than 1,000 attendees from about 350 companies.

Theme: AI on Trial

The most talked-about theme at this year’s conference was AI. In conversations with other attendees, the most common question was, "How are you using AI in your company?" and discussions were lively.

At that time, the Merpay QA team was using AI for automation and trying out various tools for other processes. So, I was most excited about the topics related to AI.

More than half of all sessions were AI-related. Even though the content differed, every session strongly emphasized "caution against AI misuse and uncertainty, rather than the efficiency and convenience AI brings." I experienced this firsthand in the first day’s tutorial, and I’d like to share it briefly.

Test by Human vs. by AI

In the first day’s tutorial, both humans and AI tested the same thing. The human testers found hidden bugs in the system, but the AI-created and executed test cases couldn’t find any bugs.

This difference comes from the critical thinking that only humans can do. When people create test cases, they first understand the specifications. If they have questions like "What changes were made?" or "What happens in specific cases?", they ask a tutor and solve them. Through this process, they remove unnecessary cases and add necessary ones to complete the test cases.

However, AI tools, no matter which one was used, just created test cases based on the given instructions and couldn’t create test cases that found bugs.

This made me realize that to be an excellent QA engineer, strong communication skills based on critical thinking is needed.

AI from a QA Perspective

Besides this tutorial, many sessions said we need to be very careful when using AI due to followed weakness;

Privacy & Security
Bias
Hallucination
Misuse by inexperienced people
Excessive automation

From the sharings so far, it might seem like the whole conference was in a negative tone for AI. However, every session assumed that AI is a helpful tool for work, so there was no anti-AI feeling.

The reason for repeatedly giving warnings is because our role is QA. Unlike other engineering roles, QA is responsible for finding problems and risks. Therefore, if we don’t treat AI with strict caution, quality could suffer.

Conclusion

I attended the conference expecting to learn best practices for AI, but in the end, I came away with challenges like "Am I a good QA engineer?" and "What can I do more to be a better QA engineer?". I’ve also been thinking about my value as a QA engineer that AI can’t replace.

However, by talking to various QA engineers from different places, I realized that everyone has the same worries, and it was very motivating.

Especially about AI, I felt again that we should use it by remembering that it’s just a useful tool, not a silver bullet.

Integration of AppIntents to a Project that uses Bazel Build System

Fri, 27 Jun 2025 07:00:50 GMT

Hey guys, it’s Cyan from the Mercoin iOS Team. This time, I would like to write about my experience integrating the new AppIntents framework to an iOS project that uses a Bazel build system. Followed by the actual implementation guide as well as a brief comparison of the new AppIntents vs the old Intents frameworks.

Firstly, what is this new AppIntents framework? It’s a new framework that serves as a replacement for the old framework Intents. With this framework, it allows users to create shortcuts in the Shortcuts app that can later be executed with Siri commands, making it an impressively useful and convenient feature.

Let’s get started!

Story Time

To begin with, we were tasked with adding an additional item to the Shortcuts app using the new AppIntents framework.

When you check tutorials online on how to use AppIntents, it is fairly straightforward:

Create a .swift file
Add import AppIntents
Create a struct that conforms to AppIntent
Run the app and you should be able to see the created AppIntent on the Shortcuts app.

It is straightforward, for a project that uses the default Xcode build system. However, for a project that uses the Bazel build system, it is a whole different story. As some of you might not yet know, Mercari’s iOS app uses Bazel. If you’re curious, you could read this article by Aoyama-san.

As Bazel doesn’t have too much documentation on the internet, we couldn’t easily find references about how to use AppIntents with Bazel.

We’ve tried everything: searching the web, using Cursor AI, using Mercari’s internal AI tool, but it took quite some time before we finally could find how to do it by referencing from this commit:

In this commit, we’ve noticed that there are 2 sample BUILD files under the test/starlark_tests directory:

/targets_under_test/ios/BUILD
/resources/BUILD

If you’re wondering what a BUILD file does, basically it’s like a configuration file of a package/module. You would add dependencies here, as well as add additional info if you’re using unit tests, app extensions, app intents, and many more. You could refer to this link if you’d like to know more about BUILD files.

Since these BUILD files are under the test folder, we thought that these could be the sample BUILD files for when you’re actually trying to integrate AppIntent into a Bazel-based project. One would be the BUILD file for the main app, and one for the module containing the AppIntent files.

We’ve checked the contents of both BUILD files, and we’ve deduced that the BUILD file that has app_intents as one of the parameters, could possibly be the sample BUILD file for the main app.

So with that said, we’ve proceeded with this:

Created a separate module from the Mercari main app, and named it MercariAppIntents
Added a BUILD file for MercariAppIntents while referencing the BUILD file from resources
Updated the BUILD file for the Mercari main app while referencing the BUILD file from targets_under_test/ios since this BUILD file contains the app_intents parameter

And then when trying to run the command that generates the Xcode project, we are faced with this error message:

Error in fail: Target '@@//Projects/Products/Mercari/Apps/MercariAppIntents:MercariAppIntents' does not depend on the AppIntents SDK framework. Found the following SDK frameworks: []

which is basically this error from Bazel’s code:

We’ve searched about this error message for hours, but it seems that no one has made any blog/writing about this error:

And yeah, asking AI didn’t help either. So, as we’ve realized that this is not something that searching through the Internet could resolve, we’ve tried some solutions by doing trial-and-error.

At first, we tried adding linkopts to the BUILD file of MercariAppIntents.

swift_library(
    name = "MercariAppIntents",
    linkopts = ["-framework,AppIntents"],
)

The reason is that the module for Widget also uses that parameter so we thought that it might work for AppIntents as well. Unfortunately, it still shows the same error.

For the second try, we tried adding linkopts to the BUILD file of the Mercari main app.

ios_application(
    name = "Mercari",
...
    linkopts = [
        "-framework,AppIntents"
    ]
)

But this also didn’t work.

After spending some more hours trying to look up the internet for some information, we gave up and just asked the Architecture team for help.

The Architecture team provided us with this solution:
Add either of these to the BUILD file of MercariAppIntents:

linkopts = [
    "-Wl,-framework,AppIntents"
],

linkopts = [
    "-framework", "AppIntents",
],

And finally, both actually resolved the error during project generation. Hooray!

However, the first one didn’t show the new AppIntents on the Shortcuts app but the second one did. During this time, I looked online on what linkopts actually is for, but I think everyone else might agree with me, Bazel’s documentation isn’t very helpful and was kinda cryptic. So, I just left it there, and just assumed that linkopts is to allow a module to import a native Apple SDK as a dependency. And that the format is as above. I just took it as a “new lesson learned for the day” and moved on.

So that will be the end of the story on how we manage to make AppIntents work on a project that uses the Bazel build system. It wasn’t smooth as we expected it to be, but with the help of multiple people (special thanks to Martin-san and Aoyama-san), we’ve managed to pull it through.

Btw, this is the thread when we were trying different approaches for this task:

Actual Implementation Guide

To actually integrate AppIntents on a project that uses Bazel, it is pretty simple.

Create a new module that will contain your AppIntent structs.

In this module’s BUILD file, you should have something like this:

swift_library(
    name = "MercariAppIntents",
    linkopts = [
        "-framework",
        "AppIntents",
    ],
)

On your main app’s BUILD file, you would need to add app_intents that should reference the new module you’ve just created, as well as add it to deps so that you would see the module when you open Xcode.

app_intents = [
    "//Projects/Products/Mercari/Apps/MercariAppIntents",
],
...
deps = [
    ...
    "//Projects/Products/Mercari/Apps/MercariAppIntents",
    ...
]

Once you’ve done that, you can proceed to generate your Xcode project.

In your new module, you can now add a file that will contain your AppIntent struct.

import AppIntents
import UIKit

internal struct YourAppIntent: AppIntent {
    // These are part of the AppIntent conformance
    static let title: LocalizedStringResource = "title"
    static let description = IntentDescription("description")

    // Set this to true so that your app will be opened when intent is run
    static let openAppWhenRun: Bool = true

    // This is also part of the AppIntent conformance
    @MainActor func perform() async throws -> some IntentResult {
        // You can put any custom URL you want here 
        await UIApplication.shared.open("yourapp://app/home")

        // Just return it like below
        return .result()
    }
}

Once you’ve run your project, you should be able to see your new intent on the Shortcuts app. And it should display the title and description we have set above.

If you would want to use Localization for your AppIntent, you can do so by creating Localizable.strings. Do note that you need to have the file to be exactly Localizable.strings as that would be the one that will be recognized by the AppIntent files.

For example, if you have set the Localizable.strings as below:

// ...en.lproj/Localizable.strings  
"title" = "Test Title";
"description" = "Test Description";

// ...ja.lproj/Localizable.strings 
"title" = "タイトル";
"description" = "内容";

You’d have something like this:

So basically, if you have a Localizable.strings on your module, the string you write here will basically be the key on your Localizable.strings file.

static let title: LocalizedStringResource = "title" // used as key
static let description = IntentDescription("description") // used as key

If you don’t have a Localizable.strings, it would just basically display that key as-is.

New AppIntents vs Old Intents

There are some key differences between the old Intents and the new AppIntents. Using the old Intents framework, to define custom intents, you would actually need to create a .intentdefinition file and input the values for your title and description on the file that looks like this:

I mean it is easy to comprehend and input values, but the problem is when you want to localize your Intent. How it is localized is done like this:

First, you have to look for the key for your title or your description. Look for your .intentdefinition file and Open As → Source Code. Once you see it like below, and take note of the key for your title or your description.

And then, on your Intents.strings, you would need to use the key above to use as key for your localization file.

"6TIN6s" = "Title";

Just from this, seeing that non-human readable key, you’d wish to move to using the new AppIntents framework already.

For AppIntents, on the other hand, as shown on the previous section, you would basically need these 2 files only:

A .swift file containing your AppIntent:

internal struct YourAppIntent: AppIntent {
    static let title: LocalizedStringResource = "title"
    static let description = IntentDescription("description")

    @MainActor
    func perform() async throws -> some IntentResult {
        ...
        return .result()
    }
}

Then a Localizable.strings, that has content like this:

"title" = "...";
"description" = "...";

With just these two, AppIntents is leaps ahead of the old Intents framework.

Additionally, you could do other stuff like AppShortcutsProvider. This is a sample code in how to use it:

import AppIntents

struct ShortcutsProvider: AppShortcutsProvider {
    static var appShortcuts: [AppShortcut] {
        AppShortcut(
            intent: SampleIntent(),
            phrases: ["Sample \(.applicationName)"],
            shortTitle: "title 1",
            systemImageName: "cup.and.saucer.fill"
        )
        AppShortcut(
            intent: SampleIntent2(),
            phrases: ["Sample 2 \(.applicationName)"],
            shortTitle: "title 2",
            systemImageName: "cup.and.saucer.fill"
        )
        AppShortcut(
            intent: SampleIntent3(),
            phrases: ["Sample 3 \(.applicationName)"],
            shortTitle: "title 3",
            systemImageName: "cup.and.saucer.fill"
        )
    }
}

Which could display something like this:

New Shortcut for the Mercari App

After the completion of this research on how to use AppIntent on a project using Bazel, we’ve successfully added a new shortcut (using the new AppIntents) for the Mercari iOS app. Previously, there was an already existing shortcut (using the old Intents SDK) from the Merpay team. This new shortcut allows users to go directly to the bitcoin chart screen, the screen which the Mercoin team mainly handles.

Setting up the shortcut with a custom name "Open MC"

You could have something like this:

As you can see from the video, with Siri Shortcuts you could also use Siri to open the shortcut by just saying the custom name you set to the shortcut.

Conclusion

So yeah, that’s it!

Hopefully, you’ve just successfully added AppIntents to your iOS project with Bazel. If you already have existing custom intents using Intents framework on your project, you could actually still see them even after you’ve added newer intents with AppIntents. It was a good thing that these two could co-exist with one another.

I wished there was a blog/resource like this when I was working on this task, but unfortunately there wasn’t so I was hoping I could be of some help to other developers who would face this problem as well.

Thank you so much for staying!

I hope you enjoyed reading this article 🙂

References

The next article will be by @hiro. Please look forward to it!

The Story Behind Mercari Design System Rebuild

Wed, 25 Jun 2025 12:00:33 GMT

This is vwxyutarooo, an Engineering Manager on the Design System team.
We have recently fully renewed the design system used for Mercari’s app and web development.
In this article, I will introduce the problems we faced with the Design System and the concepts we are using to solve them.

Background

In 2020, we began the development of the Design System alongside our major codebase renewal project called GroundUp. The Design System at this stage was called 3.0 internally.

While 3.0 might sound quite advanced, it includes versions targeting specific platforms and past versions that never saw the light of day due to one reason or another. In essence, 3.0 is the first Design System that is adopted company-wide since the previous versions (1 and 2) are developed but not adopted.

In the lifetime of our Design System of approximately 5 years, we often encountered situations where components created with 3.0 needed to handle use cases far beyond their initial intended design. As a result, many new feature developments could not be implemented using the Design System components, leading to detached symbols with modifications and many unofficial components, called “custom components”, being created internally.

Let me briefly explain why this happened using the example of a component called ItemObject in below.

This component is frequently used across multiple screens. During the 3.0 development, it was extracted as a single component designed to fit a variety of use cases. The component implementation was complex, and several unique elements were displayed or hidden using properties. Internally, we call this a polymorphic API.

As time went on, the necessary elements and required display patterns continued to increase, from further feature development after the 3.0 release.

The problem with this approach is that the number of combination patterns to consider doubles as individual UI optimizations progress.
Furthermore, as the structure deepens, such as element B or C appearing when a specific element A is displayed, the resulting Polymorphic API both increases complexity and reduces maintainability, because we’re trying to solve too many use cases with a single component.

To overcome this situation, we decided to revamp the definition of components and rebuild the Design System with a whole new design philosophy — Atomic Design.

Atomic Design Methodology

Atomic Design is a methodology introduced by Brad Frost that focuses on building web interfaces in a structured, hierarchical way. It emphasizes breaking down complex designs into fundamental elements to create scalable and maintainable design systems. This approach enhances consistency and reusability, facilitating better collaboration between designers and developers.
Atomic Design – Brad Frost

Wait, ain’t Atomic Design dead? Nope, we think it’s still alive.

Brad Frost: Is Atomic Design Dead? – Hatch Conference Berlin 2023

Since there are many explanatory articles and videos, including those by Brad Frost himself, regarding the component decomposition and design methods using Atomic Design, I will omit the details and introduce an example of how the ItemObject was constructed using the 4.0 approach.

As per the theory, each component is divided into its smallest unit of role.

The following image example was treated as one component in 3.0, but 4.0 defines it as a molecule component consisting of two atoms.

By repeating this process, it eventually becomes possible to construct higher-level parts like ItemObject from a collection of smaller, separate pieces. Keeping in mind the fundamental principle of making the UI modular and composed of parts, we provide the assembled, reusable components as molecules or organisms.

For components like ItemObject, where use cases are highly specific and fragmented, we prioritize managing highly reusable and commonly used ones as part of the Design System. On the other hand, for use cases that are less frequent or involve only subtle differences, we intentionally avoid providing complete organisms. Instead, we let users assemble these components directly within the context of their use case.

However, assembling components during use can sometimes impose a burden on users. To mitigate this, we provide examples of assembly methods in the form of "recipes" or "blueprints" to serve as supplementary resources.

We may distribute them as Molecule or Organism components, or we may leave the assembly to the user. As an intermediate step, we sometimes add examples of atom composition patterns in the documentation as recipes/blueprints and have the user assemble them.

We determine whether to create a molecule, organism, or blueprints by considering things like the frequency of component use, the context of use, or the content component dependencies.

Since Blueprints and recipes are concepts unrelated to Atomic Design, I will introduce their content in the next section.

Component Design Strategy

Atomic Design provides a framework for decomposing and constructing components of the Design System, but it does not indicate what should be a component and what kind of components should be managed as the Design System.

In our team, we used Atomic Design for the inward layer from the Design System, and we designed the outward layer independently. The following diagram roughly expresses the layer. The closer to the inside, the more it is the Design System, and the closer to the outside, the less it is the Design System. It is rare to be able to draw a strict boundary in reality, and the boundary is often a gradation, so I intentionally expressed this gradation in this diagram.

Let’s look at each layer in order:

Snowflakes

One-off components, with limited potential for reuse, due to highly specific content or use context.
Restrained use is recommended.
Custom Component
A UI component that cannot be expressed by the Design System component specs. These components are detached from symbols or modified beyond the component specs by properties that cannot be constrained on Figma, such as strokes.
Because the component specs don’t align with the Design System, either the Design System specs should be expanded to add support in the future, or the UI specifications should be adjusted to align with the Design System to thin out this layer.
Blueprint
It literally means blueprint, used in the sense of a design drawing or a completion prediction diagram.
Blueprints provide a comprehensive design drawing from Figma design data to iOS, Android, and Web source code.
It is mainly used for things that have strong content/context dependence but are frequently used, or when the assembly method is complicated, although they have a usage close to one-offs like snowflakes.
Design Recipes
Components for which design drawings are provided only in Figma design files. Not provided on source code.
For things where the need to define them as components in implementation is low, such as receiving the benefits of a framework, they are used as components only in Figma design files for design efficiency (common in layout-related components).
While Blueprint provides recipes for both design (Figma) and source code, Design Component provides only design (Figma) recipes.
Design System
Independent components that are content/context-independent and reusable.

This layer is deeply influenced by the vocabulary proposed by Brad Frost. Since this does not have an explicit name like Atomic Design, I will simply call it component vocabulary from the expression in the article.

Design system components, recipes, and snowflakes

A design organization where all UI components are completed by the Design System may be said to be the strictest design organization. It is difficult to realize, but there seem to be quite a few such organizations.

This model fits very well when seeking a slightly more rational compromise line. While allowing a certain number of content-dependent components that inevitably occur in product development as one-offs, we can manage them by giving them vocabulary and layers, and create a mindset to keep them thin. And by making things that are reusable but not sufficiently motivated to be managed as a Design System (yet) as recipes that fill the gap between the Design System and Snowflakes, we intend to optimize maintenance costs and returns by giving a gradation to the entire component layer.

Next, we will look at the design and division guidelines for Design System components. As introduced at the beginning, in the previous system, having too many behaviors and variants in one component ultimately led to decreased convenience and maintainability.

Based on these lessons learned, we focused on semantic and simple decomposition in the new system, and set the following four as component design guidelines.

Guidelines for Component Design and Segmentation

Let’s take a look at the design and segmentation guidelines for Design System components. As mentioned earlier, the previous system suffered from decreased usability and maintainability due to an overloading of behavior and variants within single components.

Building on these lessons, the new system emphasizes semantic and straightforward decomposition. The following four points have been established as the core guidelines for component design:

Semantic
"Instead of making components based on visual proximity, we define/divide components based on behavior and semantic classification and always provide consistent behavior."

For example, Mercari has a round, clickable component called a chip.

In 3.0, all were defined as one component, but even though the component looks similar in all cases, the behavior of this component is "overloaded", solving very different design problems with the same visual design pattern.

Toggle: State changes with tap.
Removable: Disappears with tap.
Text Input: Tap triggers another action.

At first glance, they may seem like just different states of a common component, but the tappable area, style on tap, and hover (Web) style are also different. Expressing these with one component requires considering unnecessary dependencies, so by splitting Chip into several components based on behavior, we can avoid the complex unnecessary dependencies introduced at the beginning and improve component maintainability.

Properties
"Can have slight visual variations based on different colors, roundness or squareness of corners. However, it cannot change the shape or behavior of the component."

In the previously introduced chip component, it has a stroke style property like solid/dotted. This is a visual variation and does not change the shape or behavior, so it does not violate the first Semantic guideline.

Optional Elements
"Components can have optional elements (such as optional icons or text)."

Child elements such as prefix/suffix icons for buttons can be added.
Care must be taken not to contradict the fourth guideline, "No polymorphic API", which will be introduced next.

No polymorphic API
"Should have a consistent API (required properties should not change based on the presence or absence of another property)."

Let’s explain using an image. The following image is a component called ItemThumbnail defined in the old Design System 3.0. In 3.0, only the Large size allowed discount or price elements, but this is considered a polymorphic API and is designed to be avoided in the new guidelines.

"Nested conditions that occur under specific conditions" ultimately lead to the complexity of management as introduced at the beginning.

Code example in Polymorphic API:

ItemThumbnail(
    size = Medium
)

ItemThumbnail(
    size = Large(
        discountPrice = 900¥,
        price = 1,000¥
    )
)

In 4.0, these problems are avoided by decomposing and reconstructing the components. An Organism component called ItemTile is prepared, and Atoms and Molecules containing ItemThumbnail as constituent elements are included.

Code example in non-polymorphic API:

ItemThumbnail(
    leftBottomContentSlot = <other atoms/molecules/organism>
)

Result
Just as a reference, our Design System based on the Atomic Design ultimately ended up consisting of about 150 components, with the component distribution as follows:

Atoms: 50
Molecules: 60
Organisms: 40

Whether this is appropriate or excessive will become clear as more and more teams and projects start using the new system.
Additionally, we were able to avoid creating complex overloaded components and polymorphic APIs. For example the ItemObject example mentioned earlier, we took an approach where the component layout is provided as ObjectLayout, and example assemblies using different parts for different use cases are offered as blueprints.

ObjectLayout:

ItemObject (blueprint):

The bloated code, which once reached about 700 lines on iOS (Swift), has been reduced to under 30 lines. While some code is still generated during the actual assembly, making it not a pure reduction, this effort helped simplify and streamline areas where the abstraction and reusability of components had previously failed.

Conclusion

Through this Design System 4.0 renewal project, we faced past challenges and gained important learnings to evolve into a more flexible and sustainable system.

From the lesson that excessive generalization of components creates complexity and significantly reduces maintainability, we returned to the principles of Atomic Design, dividing components into the smallest units, and shifted to a design that enhances reusability. This allowed each component to have a single responsibility, making changes and testing easier.

At the same time, by rethinking what components should be and rebuilding from scratch, we were able to reflect the knowledge and experience gained in 3.0 into the new system.

Currently, it is in the initial stage of operation, so mid to long-term evaluations will be conducted in the future.

Furthermore, with the automation of design and coding, including Figma AI and Figma MCP, we believe that Design System components that reflect the branding concept and have semantic meaning will increase in importance as a hub and as a provider of context for AI.

We will continue to provide updates if there are any.

Thank you for reading to the end.

Building a company-wide framework for improving DevEx in Mercari Group

Tue, 24 Jun 2025 10:00:45 GMT

This is the 17th blog post in our Merpay & Mercoin Tech Openness Month 2025 series.

I’m ntk1000, Engineering Manager overseeing both Merpay’s KYC and Partner Platform team. Today, instead of focusing on a specific team issue, I’d like to share our company-wide Engineering OKR initiative aimed at enhancing Developer Experience (DevEx).

1. Why DevEx?

Developer Experience (DevEx) refers to how smoothly and stress-free developers can work and how much they can focus on meaningful tasks.

In research proposed by Nicole Forsgren and others, "Good DevEx improves developer satisfaction and efficiency, leading to higher productivity and retention, and ultimately contributing to business success." (Reference: The SPACE of Developer Productivity)

Google also emphasizes "how much time developers can spend on truly value-generating work" and treats DevEx improvement as a key factor in product quality and speed. (Reference: How Google Measures Developer Productivity)

As such, DevEx is not just an indicator of development efficiency; it is a strategic theme directly linked to team health and product competitiveness.

With the rise of AI, business diversification, and global expansion, engineering organizations are becoming more complex. New challenges are emerging in developers’ daily work, such as difficulty in securing focus time and making autonomous decisions. As complexity increases, structural friction that cannot be resolved through individual effort or goodwill becomes more visible.

For example, the KYC and Partner Platform teams I manage play a platform role, providing common features needed by internal teams and products. Therefore, we need to pursue two goals: developing to meet the growing and global demands of services, and improving our own platform. In reality, most time and resources are consumed by the former, and improvements to the latter are delayed, creating a vicious cycle. This is a form of structural debt that cannot be solved by individuals or single teams.

That’s why we regard DevEx not as an operational or score-optimization issue, but as a strategic initiative to balance development team sustainability and product competitiveness. To empower teams to act autonomously in complex environments, the entire organization must address structural issues together. We chose a systematic, company-wide approach to DevEx improvement rather than leaving it solely to EMs and development teams.

2. Measurement as a Starting Point for Action and Dialogue

We adopted DX, a DevEx visualization tool combining qualitative survey data with delivery throughput and other quantitative data. The goal is not to generate a score, but to provide a catalyst for teams to reflect objectively on their work styles, articulate challenges, and take action for improvement. By combining qualitative and quantitative visualizations, engineers and EMs could share previously implicit challenges, which sparked constructive conversations.

To avoid “measure and forget,” we designed a quarterly improvement cycle. Surveys are not just numbers; they visualize team voices and support EMs and members in identifying and articulating underlying issues. This forms the basis for prioritizing and committing to improvements with shared understanding. Through this design, a continuous cycle of Measure → Decide → Act → Reflect has taken root.

3. Designing an Improvement Cycle That Works Across the Organization

Here’s how the improvement cycle works:

Measure: Conduct a 15-minute anonymous survey each quarter
Decide: EMs review results, discuss with the team, and identify areas to prioritize
Act: EMs create action plans and lead implementation
Reflect: Teams review the impact through retrospectives and the next survey

The core actors in this process are the engineers and EMs within each team. Meanwhile, Manager of Managers (MoM), Directors, and VPs are responsible for ensuring follow-through and resolving issues escalated by teams—so that local improvements connect to broader organizational change.

This process embodies the principle in Section 2: measurement is just a starting point for dialogue and action. The aim is not to fixate on scores, but to read between the lines of data and comments to develop feasible, meaningful improvements. The process is designed to be simple and repeatable, empowering team autonomy.

Because the cycle is quarterly and runs alongside regular work, each team is encouraged to focus on one or two areas of improvement to avoid overload and ensure execution. Specifically, EMs use vote counts, comment volume, and score gaps from company/industry benchmarks to identify top priorities through team discussions. From there, they choose realistic actions based on feasibility and consensus—not quantity.

In this initiative, we aimed to systematize DevEx improvement not as an individual or team-specific effort, but as an organizational mechanism and cultural norm. In our first cycle, we achieved 100% survey participation from engineers and 100% action plan submissions from EMs.

The following points contributed to ensuring a high participation rate:

The process was established as a cross-functional initiative aligned with the engineering organization’s overall OKRs.
We continually communicated the reasons and objectives behind focusing on DevEx improvements, not only to engineers but also to the entire organization.
During the survey period and the EM-driven improvement planning phase, we held Lunch & Learn sessions (informal learning and Q&A sessions during lunch) to increase engagement opportunities.
We organized multiple Open Door Sessions to address questions and concerns related to DX and the survey, helping alleviate doubts and uncertainties.
Before starting the process, we introduced the entire improvement cycle at an All Hands meeting to foster understanding and alignment on the significance and approach.

4. Structural Challenges Identified Across Teams

This continuous improvement cycle works not only at the team level but also as an organization-wide system for reflection and support. As a result, we were able to surface structural issues beyond the scope of individual teams.

While we cannot share internal scores, some of the most common challenges included:

Lack of Deep Work (uninterrupted time for focus): This refers to the issue of engineers not having enough time to immerse themselves in complex tasks that require deep focus. Meetings, interruptions, and unclear priorities disrupt focus. This issue received the most votes across teams. Solving complex problems requires deep focus, which is undermined by constant context switching. This is not merely a time management problem—it’s a structural issue related to how organizations operate.
Friction in Cross-Functional Collaboration: Product development does not end in Engineering; it requires smooth collaboration with Product, Legal, CS, and others. As businesses diversify and organizations scale, the number of teams and layers grows. This issue showed the largest gap compared to industry benchmarks. In our KYC and Partner Platform teams, we recognize this too—we aim to provide seamless shared capabilities, but lack of readiness and rising inbound requests are creating friction.

These issues cannot be resolved by individual EMs or teams alone. They require structural and cultural reform—such as creating rules to protect focus time, and promoting self-service systems that streamline inter-team collaboration.

5. Insights from Our Own Teams

Example: Lessons from Teams with Two Domains

Looking at results from the KYC and Partner Platform teams I manage, “Documentation” emerged as a shared issue. The KYC team also reported domain-specific issues like debugging in production and development environment complexity.

Documentation pain points include unclear sources of truth and scattered tribal knowledge, leading to repeated questions and delays in resolving specifications. This directly impacts our ability to deliver as a platform team.

Recognizing the limits of traditional documentation practices, we have already begun taking the following actions.

Building an AI/LLM-powered knowledge base using past support inquiries
Creating an internal portal where engineers can ask natural-language questions and retrieve insights from historical design docs and codebases

LLMs offer flexibility in handling fast-changing, semi-structured information. While still experimental, we believe easier knowledge access directly improves DevEx.

6. Conclusion: DevEx Is Product Experience

To build great products, we must create great environments for those who build them. DevEx isn’t just about speed or efficiency—it’s about clarity, flow, and focus.

In this first cycle, we were fortunate to see strong engagement and 100% participation across the company. That’s a great start—but the real challenge is making this sustainable, not exhausting. We want to normalize DevEx improvement as a habit.

Importantly, DevEx is not just about engineering efficiency. It’s about improving product quality and delivery. When engineers can focus, they ship better outcomes for users. That’s the mindset we’ll continue carrying forward.

We’re still learning. If you’re on a similar journey, let’s share notes.

Let’s grow better developer experiences—together.

Tomorrow’s article will be by y-arima. Please stay tuned!

Building a Flexible Checkout Solution: Frontend Architecture for Multi-Service Integration

Wed, 18 Jun 2025 10:30:45 GMT

Hello. This is David, a Frontend Engineer from Merpay Payment & Customer Platform, and EM @anzai.
This article is the 14th-day entry for the Merpay & Mercoin Tech Openness Month 2025.

This time, we would like to delve deeper into the Frontend design of CheckoutSolution, which was also mentioned in the article "New Challenges for the Payment Platform: Development of a Payment Checkout Solution" dated 2025/06/06.

New Challenges for the Payment Platform: Development of a Payment Checkout Solution（This article is written in Japanese.）

Introduction

At Mercari, we’ve been on an exciting journey to create a unified checkout solution that serves multiple services across our platform. As a Frontend Engineer on this project, I want to share our experience building a flexible, scalable checkout system that can adapt to diverse business requirements while maintaining consistency and performance.

The challenge was clear: we need to provide a purchase experience for a variety of services, including the Mercari app itself, our global-facing Mercari services, and NFT purchasing. Each service required different UI customizations, language support, and business logic, yet they all needed to share core functionality and maintain a cohesive user experience.

The Challenge: One Size Doesn’t Fit All

Traditional checkout systems are typically built for a single use case. However, our requirements were far more complex, needing to cater to:

The core Mercari app: Supporting a vast range of items and domestic transactions.
Global-facing Mercari services: Handling international sales with complex shipping and tax calculations.
NFT purchasing platforms: Managing digital product purchases with instant delivery.

Each of these areas had unique requirements:

Different UI layouts and branding needs
Multiple language support (Japanese, English, Traditional Chinese)
Varying payment methods and validation rules
Service-specific business logic and workflows

Building separate checkout systems for each service would have led to code duplication, inconsistent user experiences, and maintenance nightmares. We needed a solution that was both flexible and unified.

Technical Foundation: Building on Solid Ground

Technology Stack

Rather than reinventing the wheel, we leveraged Mercari’s established web platform standards:

React & Next.js: Following our golden path for modern web applications
TypeScript: Ensuring type safety across the entire system
Monorepo Architecture: Using PNPM workspaces for package management

The technology stack was largely predetermined by our web platform team’s tech radar, which allowed us to focus on the architectural challenges rather than technology selection.

Monorepo Strategy

One of our key architectural decisions was adopting a monorepo structure. This choice was driven by several factors:

checkout-solution/
├── packages/
│   ├── core/                 # Core elements managed by checkout team
│   ├── global-checkout/         # Global-specific implementations
│   ├── nft-checkout/    # NFT-specific implementations
│   └── shared/               # Shared utilities and types
└── apps/
    └── checkout-app/         # Main Next.js application

This structure enables:

Separation of Concerns: Each service team can work independently on their package
Code Reusability: Shared components and utilities in common packages
Version Management: Future capability for independent package versioning
Scalability: Easy addition of new services without affecting existing ones

Core Architecture: Elements-Based Design

The Element Concept

At the heart of our architecture is the concept of "Elements" – React components with enhanced capabilities:

interface Element {
  // Defined API for data access
  getData: () => CheckoutData;
  updateData: (data: Partial<CheckoutData>) => void;

  // Component implementation
  render: () => JSX.Element;
}

Elements are more than just React components. They:

Have well-defined APIs for data interaction
Can directly access and update our frontend data store
Require minimal scaffolding to integrate into the checkout flow
Maintain type safety through TypeScript definitions

Core vs Flex Elements

We categorized elements into two types:

Core Elements:

Built and maintained by the MP checkout team
Support all languages and use cases
Provide fundamental checkout functionality (payment forms, coupon discount, shipping selection, order summary)
Ensure consistency across all services

Flex Elements:

Developed by individual service teams
Customized for specific business requirements
Can override or extend core functionality
Allow for service-specific UI and business logic

This dual approach gives us the best of both worlds: consistency where it matters and flexibility where it’s needed.

Data Flow Architecture

Our data management follows a centralized store pattern:

// Simplified data flow
const CheckoutProvider = ({ children }) => {
  const [checkoutData, setCheckoutData] = useState(initialState);

  const updateData = useCallback((updates) => {
    setCheckoutData(prev => ({ ...prev, ...updates }));
  }, []);

  return (
    <CheckoutContext.Provider value={{ checkoutData, updateData }}>
      {children}
    </CheckoutContext.Provider>
  );
};

This centralized approach ensures:

Consistent data state across all elements
Easy debugging and state management
Seamless integration between core and flex elements
Reliable data persistence throughout the checkout flow

Design System Integration

Maintaining Visual Consistency

One of our biggest challenges was maintaining visual consistency while allowing customization. We addressed this through:

DS4 Integration: We implemented Mercari’s design system (DS4) as our foundation, using only default configurations initially.

Controlled Flexibility: When teams requested customizations (different text colors, font weights, etc.), we created a prop-passing system that allows flex elements to customize core elements within defined boundaries:

// Flex element can pass styling props to core elements
<CorePaymentForm 
  textColor="primary" 
  fontWeight="bold"
  // Other customizations within DS4 constraints
/>

This approach prevents visual divergence while still allowing necessary customizations.

Language and Localization: A Complex Challenge

Multi-Language Architecture

Supporting multiple languages across different services proved to be one of our most complex challenges:

Core Element Requirements:

Must support all languages (Japanese, English, Traditional Chinese)
Consistent translations across services
Centralized language management

Flex Element Flexibility:

Can support subset of languages specific to their service
Service-specific terminology and messaging
Custom localization logic

URL-Based Language Detection:

// Simplified language detection and routing
const useLanguageRouting = () => {
  const router = useRouter();
  const { locale } = router;

  useEffect(() => {
    // Validate locale against service-supported languages
    if (!supportedLocales.includes(locale)) {
      router.push(`/${defaultLocale}${router.asPath}`);
    }
  }, [locale, router]);
};

The complexity multiplies when you consider that different services support different language combinations, and we need to ensure users always see content in a supported language.

Single Domain Requirement: A Double-Edged Sword

The Challenge

One of our top-level requirements was to serve all checkout experiences from a single domain and URL structure. While this provides a seamless user experience, it significantly increased our technical complexity.

Without Single Domain (simpler approach):

Each service could have its own Next.js instance
Independent deployments and versioning
Isolated failure domains
Simpler routing and configuration

With Single Domain (our requirement):

Single Next.js instance serving all services
Shared deployment pipeline
Complex routing and service detection
Coordinated release management

Implementation Challenges

This requirement led to several technical challenges:

Package Versioning Complexity:
When one service releases an update, all services are affected, requiring:

Comprehensive regression testing across all services
Careful branching and release strategies
Cross-team coordination for deployments

Routing Complexity:
Hosting the checkout for all services on the same URL meant we couldn’t use the URL path to determine which service was being used. Instead we needed to determine the service and the related configurations based on the checkout session.

Internationalization:
As mentioned, one of our most complex challenges was dealing with multiple languages, with each service supporting a different subset of languages and with a different default. This was further complicated by working within a single Next.js instance. Care had to be taken to ensure that the language configurations for each service would not override one another.

Current State and Future Improvements

What We’ve Accomplished

✅ Flexible Architecture: Successfully serving multiple services with different requirements
✅ Type Safety: Comprehensive TypeScript implementation ensuring reliable interfaces
✅ Design System Integration: Consistent visual foundation with controlled customization
✅ Multi-Language Support: Working solution for Japanese, English, and Traditional Chinese
✅ Monorepo Structure: Scalable codebase organization for multiple teams

Areas for Improvement

🔄 Package Versioning: While our monorepo structure supports it, we haven’t fully implemented independent package versioning yet.

🔄 Documentation: Our frontend documentation covers the basics but needs expansion to fully explain all architectural decisions and patterns.

🔄 Visual Consistency Governance: As more teams implement flex elements, we need clearer guidelines and governance around visual customizations.

Lessons Learned

Complexity Management

The biggest lesson has been about managing complexity. Each requirement that seems simple in isolation can create exponential complexity when combined with others. The single domain requirement, multi-language support, and flexible UI customization each add layers of complexity that interact in unexpected ways.

Team Coordination

Building a platform used by multiple teams requires extensive coordination:

Regular sync meetings across all stakeholders
Clear documentation of decisions and rationale
Proactive communication about changes and impacts
Shared understanding of architectural principles

Flexibility vs Consistency Trade-offs

Finding the right balance between flexibility and consistency is an ongoing challenge. Too much flexibility leads to fragmented user experiences; too little flexibility prevents teams from meeting their specific business requirements.

Looking Forward

As we continue to evolve our checkout solution, we’re focusing on:

Improved Documentation: Creating comprehensive guides for teams implementing new services
Performance Optimization: Ensuring our flexible architecture doesn’t compromise on performance
Governance Framework: Establishing clear guidelines for visual and functional consistency

Conclusion

Building a flexible checkout solution for multiple services has been one of the most challenging and rewarding projects I’ve worked on at Mercari. The architecture we’ve created successfully balances the need for consistency with the requirement for flexibility, enabling teams to build service-specific experiences while maintaining a cohesive platform.

The key to our success has been:

Clear architectural principles that guide decision-making
Strong type safety that prevents integration issues
Flexible element system that accommodates diverse requirements
Extensive team coordination that keeps everyone aligned

While we still have areas to improve, our foundation is solid and scalable. As Mercari continues to expand globally and launch new services, our checkout solution is ready to support that growth.

The journey of building this system has taught us that flexibility and consistency aren’t mutually exclusive – with the right architecture and team coordination, you can achieve both.

—

Tomorrow’s article will be "@foostan’s " Challenges faced and improvements made during six years of incident response/management."
Please continue to enjoy!

SRE2.0: No LLM Metrics, No Future: Why SRE Must Grasp LLM Evaluation Now

Mon, 16 Jun 2025 22:42:01 GMT

Hello! I’m Takahiro Sato (@T), an SRE at Fintech. I’ve published this article for the 11th day of Merpay & Mercoin Tech Openness Month 2025.

Site Reliability Engineering (SRE), a form of reliability management advocated by Google and widely popularized by the Site Reliability Engineering Book, has redefined the relationship between development and operations. Starting with SLI/SLO and error budgets, it has been reinforced with metrics such as availability, latency, error rate, traffic, resource saturation, and durability.

In recent years, the progress of Large Language Models (LLMs) has been remarkable. As opportunities to use LLMs in services increase, we often encounter phenomena that are easily overlooked by conventional metrics, such as the following:

Answer quality changes after a few lines of a prompt are changed.
Hallucinations surge even when latency and error rates are good.
Answer styles drastically change with minor model updates.

In other words, to protect the "reliability of LLM services", it is becoming necessary to monitor not only classic infrastructure metrics but also LLM-specific quality metrics.

In this article, we will introduce all the procedures ranging from selecting essential metrics for evaluating the reliability of LLM services to specific measurement and evaluation methods. We will also include a demo using the DeepEval library.

1. General Evaluation Metrics for LLM Services

What metrics should we focus on to measure the reliability of LLM services? LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide lists the following representative examples of evaluation perspectives:

Metric Name	Description
Answer relevancy	Measures how appropriately the answer responds to the question.
Task completion	Measures how accurately the given task is accomplished.
Correctness	Gauges how closely the answer matches a pre-prepared correct answer
Hallucination	Gauges whether the content includes factually incorrect or fabricated information
Tool correctness	Gauges whether the correct tool was selected and executed to achieve the task
Contextual relevancy	Gauges how appropriate the searched information is for the question
Responsible metrics	Gauges whether the content includes discriminatory or offensive expressions, or whether it is biased towards specific attributes
Task-specific metrics	Gauges the performance of LLMs in "specific tasks" such as summarization or translation

By monitoring infrastructure SLIs such as availability and latency, which are typical metrics for conventional services, we have been able to understand customer satisfaction levels in relation to the user journey. However, with LLM services, the quality of generation itself, such as whether a response is in line with the user’s intent and based on facts and whether the task has been completed correctly, directly affects customer satisfaction. Therefore, in addition to conventional SLIs such as availability and latency, it is necessary to design SLIs that capture the unique generation quality of LLM services and to establish a metric system that can quantitatively show whether customers can quickly obtain the correct answer as intended. So, when designing metrics for LLM services, which metrics should be selected specifically?

1.1. Pitfalls of General Evaluation Metrics

General evaluation perspectives such as answer relevance, correctness, and the presence or absence of hallucinations, as shown in the table above, constitute a framework, but they may not be able to equal the unique success conditions of all LLM service use cases. For example, without unique metrics such as comprehensiveness and absence of contradictions for summarization services, or "relevance of search context" for RAG, it is often impossible to fully measure the value that users receive. The article The Accuracy Trap: Why Your Model’s 90% Might Mean Nothing explains that although a customer churn prediction model achieved 92% accuracy rate during testing, in practice, it generated false positives and caused oversights that resulted in an increased churn rate.

The lesson here seems to be this: Prioritize end-to-end evaluations from the user’s perspective. LLM services have complex internal structures such as RAG and agent mechanisms, but no matter how much the intermediate components are improved, the ROI will not increase unless the answers that users receive improve. The evaluation metric for whether or not to select an LLM service should measure the final output of a system as a black box and measure its results end to end. In doing so, it should also look at whether the performance correlates with such things as reduced support time and improved sales.

1.2. What Makes a Good Evaluation Metric?

The Complete LLM Evaluation Playbook: How To Run LLM Evals That Matter lists the following three conditions for excellent evaluation metrics:

Quantitative
- It must be possible to calculate a numerical score as an evaluation result. If the result can be evaluated numerically, it is desirable to be able to set a threshold that serves as a passing line or to measure the effect of model improvements by tracking changes in the score over time.
Reliable
- It must be possible to obtain consistently stable evaluation results. Given that LLM output fluctuates unpredictably, it would be problematic if the evaluation metrics were also unstable. For example, although evaluation methods using LLMs (such as LLM-as-a-judge, described later) are more accurate than conventional methods, they tend to have more variability in the evaluation results, so caution is required.
Accurate
- It must be possible to accurately reflect the performance of the LLM model with criteria that is nearly the same as actual human evaluation. Ideally, an output with a high evaluation score reflects an output that a human user would feel comfortable with. For that reason, it is necessary to evaluate output using criteria that match human expectations.

Also, no matter how high an evaluation metric value is, if it does not lead to business results such as sales and customer satisfaction it is meaningless. The article calls this metric-outcome fit (MOF) and explains that 95% of LLM metric evaluations performed in the field do not have this connection and do not create value." The article goes on to state that the only way to avoid using the wrong metrics is to keep confirming and adjusting whether the metrics can reliably determine that cases that are considered good results in business are actually favorable.

2. Overall Picture of Metric Evaluation Methods

In this next section, we will introduce the types of methods for actually evaluating metrics. There are roughly four types, and each has its own advantages and disadvantages.

Statistical methods (string-based, n-gram based, and surface base)
Methods using models other than LLMs (classifier, learned metrics, and small-LM metrics)
Hybrid methods that use statistical methods and models other than LLMs simultaneously (embedding-based metrics)
Methods using the LLM itself (LLM based and generative evaluator)

2.1 Statistical Methods

A statistical method compares the correct answer data created manually with the output text at the string level, measuring the level of similarity, and evaluating the result.

BLEU
- It assigns a score calculated by averaging the 1- to 4-gram precision between the model’s output and the expected reference translation. This precision-based score is then multiplied by a brevity penalty, which also incorporates a penalty for discrepancies in length (being either too long or too short).
ROUGE
- ROUGE-L is often used for summary evaluation. It calculates the F1 score based on LCS (longest common subsequence) for recall and precision, while ROUGE-1/2 measures how well the summary covers the original document based on n-gram recall.
METEOR
- This metric evaluates both accuracy and recall.It takes into account differences in word order and synonym matching. (The final score is calculated by multiplying the harmonic mean of accuracy and recall by a word order penalty.)
Edit distance or Levenshtein distance (available only in Japanese)
- This metric measures the difference between the output and a correct string. In practice, it is rarely used as is for comparing multiple sentence lengths, and is not used much considering the catch-up cost.

ref: LLM evaluation metrics — BLEU, ROGUE and METEOR explained

These statistical indicators are simple to calculate and have high reproducibility (consistency), but they do not consider the meaning or context of the text, so they are not suitable for evaluating long-form answers or outputs that require advanced reasoning generated by LLMs. In fact, pure statistical methods cannot evaluate the logical consistency or correctness of the meaning of the output, and the accuracy is said to be insufficient for complex outputs.

2.2. Methods Using Models Other Than LLMs

This is an evaluation method that uses machine learning models dedicated to evaluation, such as classification models and embedding models, and relatively lightweight natural language processing models.

NLI (Natural Language Inference) model
- You can classify whether the output of the LLM is consistent (entailment), contradictory (contradiction), or irrelevant (neutral) to the given reference text (such as factual information). In this case, the output score of the model is the probability value of how logically consistent a text is from 0.0 to 1.0.
Dedicated model trained based on transformer-type language models (such as NLI and BLEURT)
- This is a method of scoring and measuring the similarity between the output of the LLM and the expected correct answer. With model-based methods, it is possible to evaluate the meaning of the text to some extent , but because the evaluation model itself has uncertainty, the consistency (stability) of the score is lacking. For example, it has been pointed out that NLI models cannot make good judgments if the input sentence is long, and that BLEURT is affected by the bias of the training data and the evaluation may be biased.

2.3. Hybrid Methods That Use Statistical Methods and Models Other Than LLMs Simultaneously

These are methods positioned in the middle of the above methods that perform evaluations by combining a value embedded and vectorized by a pre-trained language model with statistical distance calculation.

Bidirectional encoder representations from transformers (BERT) Score
- Calculates the cosine similarity (available only in Japanese) between the context vectors of each word obtained by BERT, etc., and measures the semantic overlap between the output sentence and the reference sentence.
MoverScore
- Creates a distribution using word embeddings for each of the output sentence and the reference sentence, and calculates the Earth Mover’s Distance (Optimal Transport Distance) (available only in Japanese) from there to measure the difference between the two.

These methods are superior to BLEU and other statistical methods in that they can capture semantic closeness beyond the word level and surface level, but they have the weakness that they are ultimately affected by the performance and bias of the original embedding model (BERT, etc.). For example, if the pre-training model does not have an appropriate vector representation for the context of a specialized field or the latest knowledge, accurate evaluation is not possible. There is also a risk that the social bias included in the evaluation model will manifest in the score.

2.4. Methods Using LLMs (LLM-as-a-judge)

Among all the evaluation methods now available, LLM-as-a-judge has been attracting attention in recent years. This is a method where the LLM itself measures and evaluates quality of output. This approach gives advanced LLMs instructions such as "Please evaluate whether the given answer meets the criteria" and extracts evaluation scores and judgments from the model. LLMs can understand the meaning of sentences and make complex judgments, and so the major advantage is that they can automate evaluations close to human subjectivity. In fact, in the G-Eval method, which uses GPT-4 as an evaluator, the correlation between the evaluation score and human evaluation is greatly improved compared to conventional automatic evaluations, as those described in the article G-Eval Simply Explained: LLM-as-a-Judge for LLM Evaluation. On the other hand, LLM-based evaluations have issues with score stability (reliability) because the results can fluctuate depending on the response of the model. There is no guarantee that the same score will be obtained every time, even if the LLM re-evaluates the same answer, because the random elements of the model and the fluctuations in the output also affect the evaluation results.

Here are some of the typical methods of LLM-as-a-judge:

G-Eval
- A mechanism that scores evaluation criteria on a scale of 1–5. The LLM returns the evaluation score and the reason for the evaluation result (the result of chain of thought).
- QAG Score
- Automatically generates QA (yes, no, or unknown) from the output, solves the same QA in the original text, and scores the match rate between the two.
- SelfCheckGPT
- Samples N times with the same prompt, and estimates the factuality by measuring the consistency between the generated sentences (e.g., multiple comparison modes such as N-gram, QA, BERTScore). The greater the variation, the higher the possibility of hallucinations.
- DAG(deep acyclic graph)
- A decision tree type metric provided by DeepEval. Each node is an LLM judgment (yes or no). Since a fixed score is returned depending on the route, the LLM-as-a-judge is bundled with Boolean judgment nodes in a decision tree, and the partial points are deterministic.
- Prometheus2 Model
- An evaluation model of 7B/8x7B distilled from feedback from high-quality judges including GPT-4 and numerous evaluation traces. Proven with a match rate of 0.6-0.7 with humans/GPT-4 (direct scoring), 72–85% (pairwise comparison).

The following table summarizes the measurement and evaluation methods of the indicators discussed so far.

Type	Specific Method	Advantages	Disadvantages
Statistical Methods	BLEU, ROUGE, METEOR, and Edit Distance (Levenshtein Distance)	– Provides simple and fast calculation – Features high reproducibility – Requires no additional learning and is easy to implement	– Evaluates only surface matches without considering meaning or context – Not suitable for output that requires logical consistency or advanced reasoning
Methods Using Models Other Than LLMs	NLI (Natural Language Inference) Model, BLEURT, Transformer-Based Dedicated Evaluation Model	– Can evaluate meaning, understanding, and logical consistency to some extent – Offers lower calculation costs than LLMs, and can be fine-tuned independently	– Depends on the uncertainty and bias of the evaluation model itself – Accuracy tends to decrease for long sentences and content on specialized fields
Hybrid Methods	BERTScore and MoverScore	– Captures semantic closeness with embeddings and offers higher accuracy than statistical indicators – Deterministic and easily maintains reproducibility	– Depends on the learning range and bias of the embedding source model – Difficult to adapt to the latest knowledge or narrow specialized fields
Methods Using LLMs (LLM-as-a-judge)	G-Eval, QAG Score, SelfCheckGPT, DAG (Deep Acyclic Graph), and Prometheus2 Model	– Can automate complex judgments that closely resemble human evaluation – Can evaluate multifaceted quality of answers in one go	– Output is probabilistic and scores tend to fluctuate – High model usage cost and sensitive to prompts

To actually measure and evaluate these evaluation methods, requires a tool to measure them efficiently. Therefore, in this next section, we will introduce DeepEval, which I glimpsed in a reference article in the LLM evaluation libraries.

3. DeepEval

DeepEval is a Python library for evaluating LLM services. It provides a framework for creating test cases, defining evaluation metrics, and running evaluations. DeepEval supports metrics that evaluate various aspects such as response relevance, fidelity, and contextual accuracy, and also supports custom metrics, automatic generation of evaluation datasets, and integration with test frameworks such as Pytest. The official documentation provides detailed installation instructions, as well as instructions on basic usage, how to set various evaluation metrics, how to create custom metrics, and more.

Now, let’s look at the practical application of evaluation procedures based on a simple summarization service.

3.1. Practical Example: Determining Metrics and Measurement Methods for Summarization Services

Our assumption is that the summarization service discussed here receives long texts such as articles and documents as input and generates a summary of the content. I believe this is the first service people envision as a specialty of the LLM mechanism. In the following sections, I would like to envision a service that summarizes Grimm’s Fairy Tales and summarizes them into sentences simple enough for even children to understand.

3.2. Selection of Indicators

From the perspective of summarization, the indicators that come to mind as general evaluation indicators are Answer Relevancy, Correctness, and Hallucination. You can use DeepEval’s G-Eval to support the above three indicators, but it is necessary to investigate whether this case corresponds to “1.2. What Makes a Good Evaluation Metric?”"

Quantitative
- G-Eval returns a continuous score from 0 to 1, so it can be said that a numerical score can be calculated as an evaluation result.
Reliable
- G-Eval is originally probabilistic, but if you execute the following three points you can almost reproduce the same score with the same input: (1) Call the temperature option passed to the LLM model with 0, (2) fix evaluation_steps and skip CoT generation processing, and (3) specify the Rubric to make the evaluation score constant. This will allow you to always get stable evaluation results. (Strictly speaking, sampling noise and system randomness on the OpenAI side remain, so complete reproduction is not possible. We recommend using an API/backend where top_p=0 and seed can be fixed, or ultimately using majority vote/ensemble evaluation.)
Accurate
- G-Eval features evaluation with references (i.e., expected_output; in this case, the original text of Grimm’s Fairy Tales and correct answer data). It has been shown in both papers and actual operation that G-Eval has a high correlation with human judgment in tasks that focus on fact verification.

In light of the above, it seems appropriate to use DeepEval’s G-Eval for the metric evaluation of the Answer Relevancy, Correctness, and Hallucination metrics.

3.3. Decomposition of Evaluation Perspectives

In this next section, we will list the perspectives and steps necessary for evaluating the picked-up indicators and the sort of procedures in which they should be evaluated. Fortunately, there was a document from Google Cloud, Vertex AI documentation – Metric prompt templates for model-based evaluation, which seemed to be helpful in decomposing the evaluation perspectives, so this time I would like to refer to it.

Answer Relevancy
- STEP1. Identify user intent – List the explicit and implicit requirements in the prompt.
- STEP2. Extract answer points – Summarize the key claims or pieces of information in the response.
- STEP3. Check coverage – Map answer points to each requirement; note any gaps.
- STEP4. Detect off-topic content – Flag irrelevant or distracting segments.
- STEP5. Assign score – Choose 1-5 from the rubric and briefly justify the choice.
Correctness
- STEP1. Review reference answer (ground truth).
- STEP2. Isolate factual claims in the model response.
- STEP3. Cross-check each claim against the reference or authoritative sources.
- STEP4. Record discrepancies – classify as omissions, factual errors, or contradictions.
- STEP5. Assign score using the rubric, citing the most significant discrepancies.
Hallucination
- STEP1. Highlight factual statements – names, dates, statistics, citations, etc.
- STEP2. Compare the result with the provided context and known reliable data.
- STEP3. Label claims as verified, unverifiable, or false.
- STEP4. Estimate hallucination impact – proportion and importance of unsupported content.
- STEP5. Assign score following the rubric and list specific hallucinated elements.

3.4. Calculating Evaluation Scores

Now, let’s actually conduct evaluation measurements and calculate evaluation scores. First, we’ll prepare the material to be summarized and the prompt. This time, we’ll use the original text of Little Red Riding Hood from Grimm’s Fairy Tales and prepare the following prompt:

Please create a summary of the following Grimm's Fairy Tale content.

Requirements:

1. Identify and include major characters and important elements
2. Logically organize the flow of content
3. Include important events and turning points
4. Be faithful to the original text content
5. Keep the summary within 500 characters

Grimm's Fairy Tale content: {Little Red Riding Hood original text}

Summary: """

The evaluation script used is as follows:

import asyncio
import openai
from deepeval.metrics.g_eval.g_eval import GEval
from deepeval.metrics.g_eval.utils import Rubric
from deepeval.test_case.llm_test_case import LLMTestCase, LLMTestCaseParams

async def evaluate_comprehensive_metrics(client: openai.AsyncOpenAI, test_case: LLMTestCase, prompt_name: str, original_text: str) -> dict:
    """Execute G-Eval metrics evaluation"""

    # Answer Relevancy evaluation
    geval_answer_relevancy = GEval(
        name="Answer Relevancy",
        evaluation_steps=[
            "STEP1. **Identify user intent** – List the explicit and implicit requirements in the prompt.",
            "STEP2. **Extract answer points** – Summarize the key claims or pieces of information in the response.",
            "STEP3. **Check coverage** – Map answer points to each requirement; note any gaps.",
            "STEP4. **Detect off-topic content** – Flag irrelevant or distracting segments.",
            "STEP5. **Assign score** – Choose 1-5 from the rubric and briefly justify the choice.",
        ],
        rubric=[
            Rubric(score_range=(0, 2), expected_outcome="Largely unrelated or fails to answer the question at all."),
            Rubric(score_range=(3, 4), expected_outcome="Misunderstands the main intent or covers it only marginally; most content is off-topic."),
            Rubric(score_range=(5, 6), expected_outcome="Answers the question only partially or dilutes focus with surrounding details; relevance is acceptable but not strong."),
            Rubric(score_range=(7, 8), expected_outcome="Covers all major points; minor omissions or slight digressions that don't harm overall relevance."),
            Rubric(score_range=(9, 10), expected_outcome="Fully addresses every aspect of the user question; no missing or extraneous information and a clear, logical focus."),
        ],
        evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.RETRIEVAL_CONTEXT],
        model="gpt-4o"
    )

    # Correctness
    geval_correctness = GEval(
        name="Correctness",
        evaluation_steps=[
            "STEP1. **Review reference answer** (ground truth).",
            "STEP2. **Isolate factual claims** in the model response.",
            "STEP3. **Cross-check** each claim against the reference or authoritative sources.",
            "STEP4. **Record discrepancies** – classify as omissions, factual errors, or contradictions.",
            "STEP5. **Assign score** using the rubric, citing the most significant discrepancies.",
        ],
        rubric=[
            Rubric(score_range=(0, 2), expected_outcome="Nearly everything is incorrect or contradictory to the reference."),
            Rubric(score_range=(3, 4), expected_outcome="Substantial divergence from the reference; multiple errors but some truths remain."),
            Rubric(score_range=(5, 6), expected_outcome="Partially correct; at least one important element is wrong or missing."),
            Rubric(score_range=(7, 8), expected_outcome="Main facts are correct; only minor inaccuracies or ambiguities."),
            Rubric(score_range=(9, 10), expected_outcome="All statements align perfectly with the provided ground-truth reference or verifiable facts; zero errors.")
        ],
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.RETRIEVAL_CONTEXT],
        model="gpt-4o"
    )

    # Hallucination
    geval_hallucination = GEval(
        name="Hallucination",
        evaluation_steps=[
            "STEP1. **Highlight factual statements** – names, dates, statistics, citations, etc.",
            "STEP2. **Compare with provided context** and known reliable data.",
            "STEP3. **Label claims** as verified, unverifiable, or false.",
            "STEP4. **Estimate hallucination impact** – proportion and importance of unsupported content.",
            "STEP5. **Assign score** following the rubric and list specific hallucinated elements.",
        ],
        rubric=[
            Rubric(score_range=(0, 2), expected_outcome="Response is dominated by fabricated or clearly false content."),
            Rubric(score_range=(3, 4), expected_outcome="Key parts rely on invented or unverifiable information."),
            Rubric(score_range=(5, 6), expected_outcome="Some unverified or source-less details appear, but core content is factual."),
            Rubric(score_range=(7, 8), expected_outcome="Contains minor speculative language that remains verifiable or harmless."),
            Rubric(score_range=(9, 10), expected_outcome="All content is grounded in the given context or universally accepted facts; no unsupported claims.")
        ],
        evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.RETRIEVAL_CONTEXT],
        model="gpt-4o"
    )

    await asyncio.to_thread(geval_answer_relevancy.measure, test_case)
    await asyncio.to_thread(geval_correctness.measure, test_case)
    await asyncio.to_thread(geval_hallucination.measure, test_case)

    # Function to estimate rubric score (for display purposes)
    def extract_rubric_score_from_normalized(normalized_score, rubric_list):
        """Identify rubric range from normalized score (0.0-1.0)"""
        scaled_score = normalized_score * 10

        for rubric_item in rubric_list:
            score_range = rubric_item.score_range
            if score_range[0] <= scaled_score <= score_range[1]:
                return {
                    'scaled_score': scaled_score,
                    'rubric_range': score_range,
                    'expected_outcome': rubric_item.expected_outcome
                }
        return None

    answer_relevancy_rubric_info = extract_rubric_score_from_normalized(
        geval_answer_relevancy.score, geval_answer_relevancy.rubric
    )
    correctness_rubric_info = extract_rubric_score_from_normalized(
        geval_correctness.score, geval_correctness.rubric
    )
    hallucination_rubric_info = extract_rubric_score_from_normalized(
        geval_hallucination.score, geval_hallucination.rubric
    )

    return {
        "answer_relevancy_score": geval_answer_relevancy.score,
        "answer_relevancy_rubric_info": answer_relevancy_rubric_info,
        "answer_relevancy_reason": geval_answer_relevancy.reason,
        "correctness_score": geval_correctness.score,
        "correctness_rubric_info": correctness_rubric_info,
        "correctness_reason": geval_correctness.reason,
        "hallucination_score": geval_hallucination.score,
        "hallucination_rubric_info": hallucination_rubric_info,
        "hallucination_reason": geval_hallucination.reason,
    }

async def generate_summary(client: openai.AsyncOpenAI, prompt_template: str, full_story: str, model: str = "gpt-4o") -> str:
    """Generate summary using LLM"""
    prompt = prompt_template.format(context=full_story)

    try:
        response = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=300,
            temperature=0.0, top_p=0, logit_bias={}
        )
        content = response.choices[0].message.content
        return content.strip() if content else ""
    except Exception as e:
        return f"Error: {str(e)}"

async def process_prompt(client: openai.AsyncOpenAI, prompt_info: dict, full_story: str, context: list) -> dict:
    model = prompt_info.get("model", "gpt-4o")

    # Generate summary
    summary = await generate_summary(client, prompt_info["template"], full_story, model)

    # Create test case
    test_case = LLMTestCase(
        input=prompt_info["template"],  # Prompt
        actual_output=summary,  # Summary result
        retrieval_context=context  # Original text of the fairy tale to be summarized
    )

    # Execute evaluation
    metrics_result = await evaluate_comprehensive_metrics(client, test_case, prompt_info['name'], full_story)

    return {
        "prompt_name": prompt_info['name'],
        "model": model,
        "summary": summary,
        **metrics_result
    }

async def main():
    # Load the original fairy tale text
    with open('little_red_riding_hood.txt', 'r', encoding='utf-8') as f:
        full_story = f.read().strip()

    context = [full_story]

    prompts = [
        {
            "name": "prompt-01",
            "template": """Please create a summary of the following `story`.

Requirements:

1. Identify and include major characters and important elements
2. Logically organize the flow of content
3. Include important events and turning points
4. Be faithful to the original text content
5. Keep the summary within 500 characters

story: {context}

Summary:""",
            "model": "gpt-4o"
        },
    ]

    async with openai.AsyncOpenAI() as client:
        tasks = [
            process_prompt(client, prompt_info, full_story, context)
            for prompt_info in prompts
        ]

        all_results = await asyncio.gather(*tasks)

    # Result display processing
    ...

if __name__ == "__main__":
    asyncio.run(main())

The executed summary result was as follows:

Once upon a time, there was a lovely little girl called Little Red Riding Hood. She received a red hood from her grandmother and always wore it.
One day, she went through the forest to her grandmother's house to deliver sweets and wine to her sick grandmother.
On the way, she met a wolf and told him where she was going. The wolf went ahead and swallowed the grandmother, then deceived Little Red Riding Hood and swallowed her too.
However, a hunter who was passing by cut open the wolf's belly and rescued Little Red Riding Hood and her grandmother. Little Red Riding Hood learned a lesson and vowed never to stray from the path in the forest again.

The results evaluated by G-Eval are as follows (excerpt from the first run):

- Answer Relevancy: 0.912
  - Expected Outcome: Fully addresses every aspect of the user question; no missing or extraneous information and a clear, logical focus.
  - Reason: The summary includes key characters like Little Red Riding Hood, her grandmother, the wolf, and the hunter. It logically organizes the flow of events, such as the journey through the forest, the encounter with the wolf, and the rescue. Important events like the wolf's deception and the rescue by the hunter are covered. The summary is faithful to the original text and concise, with no extraneous information.
- Correctness: 0.901
  - Expected Outcome: All statements align perfectly with the provided ground-truth reference or verifiable facts; zero errors.
  - Reason: The main facts in the Actual Output align well with the Retrieval Context, including the characters, events, and moral of the story. Minor details like the specific dialogue and actions are slightly condensed but do not affect the overall accuracy.
- Hallucination: 0.903
  - Expected Outcome: All content is grounded in the given context or universally accepted facts; no unsupported claims.
  - Reason: The output closely follows the context with accurate details about Little Red Riding Hood, her grandmother, the wolf, and the hunter. The sequence of events and character actions are consistent with the context, with no unsupported claims.

Looking at the evaluation reasons that determined the scores, it appears that each indicator is being evaluated appropriately. As introduced in 3.2 Selection of Indicators, G-Eval experiences evaluation fluctuations. Therefore, we executed the above script 50 times. The scatter plot of the measured evaluation values is shown below.

As a result, all indicators achieved scores of approximately 0.9 or higher, but would it be possible to set the SLI value for each indicator to approximately 0.9 and to set the SLO to 0.9 or higher as a target value?

3.5. Review of Evaluation Metrics

As introduced above, this service summarizes Grimm’s Fairy Tales and summarizes them in sentences simple enough for even children to understand. To make the above summary results understandable for children, we should also consider the following indicators:

Readability: Are there difficult kanji characters (words) or expressions that children cannot read?
- "deceived"?, "lesson"?, "wine"? (The Japanese version of the summary used old expressions and difficult kanji)
Safety/Toxicity: Are there expressions that, when compared with modern compliance, are too violent for children?
- E.g., cut open the belly

It is necessary to select evaluation indicators with an awareness of closely linking them to customer value and business KPIs. In the case of this summarization service, rather than general evaluation indicators, the above indicators should be prioritized as task-specific metrics considering the target audience. Accordingly, the prompt would also need to be modified.

That said, it is difficult to create a perfect set of indicators on the first attempt. The Complete LLM Evaluation Playbook: How To Run LLM Evals That Matter states that it is desirable to start with one evaluation indicator and eventually narrow it down to five. It is necessary to select, measure, and evaluate indicators while being aware of how much the evaluation indicator scores match the metric outcome fit—the connection between indicators and outcomes (frequent use by children).

(In the case of an actual service, as a business KPI, providing images rather than text might yield better results)

3.6. Exploring Automation Possibilities

In the following example, humans performed indicator selection, evaluation score calculation, and indicator evaluation review. G-Eval then uses a mechanism that makes GPT-4 class models decompose and think about evaluation procedures themselves and return only the final score. In this way it can automate evaluation criteria application, scoring, and aggregation in one step in place of a human operator. Here is an example of that procedure:

Present the evaluation tasks: Give the LLM used for evaluation a task explanation such as "Please score the generated text that will be presented according to certain evaluation criteria on a scale of 1 to 5." When performing this step, clearly indicate the definition of the evaluation criteria and teach the LLM the context of the task (for example, present the indicator list that was in the general evaluation indicators for LLM services).
Decompose the evaluation perspectives: For the indicators selected by the LLM in 1., have the model list the necessary perspectives and steps by itself.
Calculate the score: Next, have the model evaluate the actual input and output it according to the evaluation steps generated earlier.

As a point of caution, when LLMs act as evaluators, they tend to overestimate LLM-like outputs and have vulnerabilities where scores can be manipulated with the insertion of just a few words. Even if we try to mitigate this through evaluation with a different series of LLM models, complete neutrality cannot be guaranteed, for such things as pairwise comparison where two answers are compared side by side, or anomaly detection. Also, as introduced in 3.2 Selection of Indicators, G-Eval has reproducibility issues where evaluation fluctuates for the same answer due to its probabilistic evaluation method, requiring measures such as fixing evaluation prompts and seeds. For these reasons, it is essential to take a two-stage approach where human review is always used in conjunction for correction and verification of final judgments.

4. Summary

In this article, we introduced a range of topics from selecting essential metrics for evaluating the reliability of LLM services to specific measurement and evaluation methods and included demonstrations using the DeepEval library. How to define metrics for LLM service reliability evaluation as SLIs, which cannot be fully measured by conventional metrics such as availability and latency alone, is a new field for SRE as well. The approach of using evaluation tools such as DeepEval, which we tested for this article, is just one of many options. The field of LLM evaluation metrics is still under active research, and there seems to be no single correct answer yet to the question of how to measure the reliability of LLM services. However, even if new evaluation metrics and new measurement methods are discovered in the future, I believe that one fundamental question will remain unchanged: Do these metrics really represent customer satisfaction? Along with technological progress, I hope we can continue to engage in daily SRE work without forgetting this question.

Tomorrow’s article will be “AI Hackathon at Mercari Mobile Dev Offsite” by @k_kinukawa san. Stay tuned!

References

Site Reliability Engineering Book: https://sre.google/books/
LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide: https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation
The Accuracy Trap: Why Your Model’s 90% Might Mean Nothing: https://medium.com/%40edgar_muyale/the-accuracy-trap-why-your-models-90-might-mean-nothing-f3243fce6fe8
The Complete LLM Evaluation Playbook: How To Run LLM Evals That Matter: https://www.confident-ai.com/blog/the-ultimate-llm-evaluation-playbook
Levenshtein Distance: https://note.com/noa813/n/nb7ffd5a8f5e9
LLM evaluation metrics — BLEU, ROUGE and METEOR explained: https://avinashselvam.medium.com/llm-evaluation-metrics-bleu-rogue-and-meteor-explained-a5d2b129e87f
BERTScore: https://openreview.net/pdf?id=SkeHuCVFDr
BERT: https://en.wikipedia.org/wiki/BERT_(language_model)
Cosine Similarity: https://atmarkit.itmedia.co.jp/ait/articles/2112/08/news020.html
MoverScore: https://arxiv.org/abs/1909.02622
Earth Mover’s Distance (Optimal Transport Distance): https://zenn.dev/derwind/articles/dwd-optimal-transport01#%E6%9C%80%E9%81%A9%E8%BC%B8%E9%80%81%E8%B7%9D%E9%9B%A2
G-Eval (Paper): https://arxiv.org/abs/2303.16634
G-Eval Simply Explained: LLM-as-a-Judge for LLM Evaluation: https://www.confident-ai.com/blog/g-eval-the-definitive-guide
QAG Score: https://arxiv.org/abs/2210.04320
SelfCheckGPT: https://arxiv.org/abs/2303.08896
DAG (deep acyclic graph): https://deepeval.com/docs/metrics-dag
Prometheus2 Model: https://arxiv.org/abs/2405.01535
DeepEval: https://deepeval.com/docs/getting-started
Vertex AI – Metric Prompt Templates for Model-Based Evaluation: https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templates
Little Red Riding Hood: https://ja.wikipedia.org/wiki/%E8%B5%A4%E3%81%9A%E3%81%8D%E3%82%93

Rethink Tool’s UI/UX – Human-Centric to AI-Driven

Tue, 03 Jun 2025 10:00:27 GMT

This post is for Day 2 of Merpay & Mercoin Tech Openness Month 2025, brought to you by @ben.hsieh from the Merpay Growth Platform Frontend Team.

Merpay Growth Platform develops an internal platform for Mercari’s user engagement and CRM activities, empowering marketing users.
This article introduces our efforts to evolve our internal platform driven by AI.

Background

For approximately four years, the Merpay Growth Platform has developed an internal platform called Engagement Platform. Previously, Mercari had disparate tools and services addressing similar problems independently for various use cases, leading to redundancy.

To address fragmented processes and diverse use cases, the Engagement Platform was developed as a unified solution. This necessitates close collaboration with marketing teams to understand their specific needs and deliver a flexible solution capable of handling a wide variety of applications.

The Role of Frontend Team

Building internal systems might seem easier because they have fewer users. However, the Growth Platform Frontend Team has been quite ambitious over the past few years, developing our internal platform into a full-fledged CMS and CRM admin dashboard.

This means it’s a full-stack operation, requiring us to address both the UI/UX of the admin tools and the challenges of the content service to handle Mercari’s extensive user activity in the production environment. To know more about this team’s interesting initiatives, check our posts previous below:

Significance & Challenges of Admin System UX

Internal tools often get the short end of the stick when it comes to good design. But our team is determined to change that. We’re aiming to build an internal platform with a really polished, user-friendly feel – like something you’d see in a real product.

That means tackling the tricky bits of both our admin tools and content systems, so our marketing folks have a smooth experience even with tons of user activity. The ultimate goal is to help empower non-engineers to have entire control over their operations, bring their ideas to life.

Therefore, the team must prioritize ease-of-use when implementing minor features. Design language should be employed to simplify complex engineering concepts, making them understandable to a broader audience. User experience is more crucial than we ever imagined!

Engagement Platform is now an intricate system that manages user segmentation, incentives, notifications, and content. Ensuring a clear and collaborative user experience across these interconnected resources and functionalities is challenging.

💭 Consider a typical scenario: a promotion triggers emails and push notifications containing links to content within the platform. How can we effectively guarantee consistency in messaging across all these touchpoints?

The team is working on complex real-world applications and developing assistive tools to ensure consistency across diverse resources and streamline their alignment. However, this approach faces inefficiencies due to:

The tension between the specificity required for consistency and the need for flexibility.
The limitations of static analysis in identifying all inconsistencies, particularly in natural language information. Also, these static analysis tools introduce maintenance effort per use case, not very scalable and increase overhead over time.

These are tradeoffs the team continuously takes into consideration. With rapid growth of our business needs, the development effort to support it also scales rapidly as these all need engineers’ hands-on effort.

For example, introducing a new platform capability to users, usually involves several steps:

Backend Service Readiness: The backend service must be developed to handle business logic and offer APIs for client-side interaction.
Client-side Development and UX Design: This involves working with the product team to define the user experience and then implementing the necessary UI modifications within the application to make the functionality accessible to users.

Instead of making engineers build every little thing and cluttering the interface with a million buttons, wouldn’t it be cool if our tools could just talk to us?

Agentic UX: Let’s Make Our Tools “Talk”

So, yeah, Language Models (LLMs) are looking pretty tempting these days. The fact that they can actually understand what we’re saying is a definite plus. And hey, let’s be real, playing around with this new tech sounds kinda fun, right? 😄

Think about all those AI apps popping up that everyone’s using. Notice a pattern? It’s usually some kind of chat thing going on.

"Why Chat?"

Basically, "talking" to an LLM is like asking it for information using normal language. One of the cool things about this kind of interaction is that we don’t need to make a bunch of changes to how our tools look to add new stuff.

"The key is still how to efficiently and precisely let our users access what our service can do."

Remember when LLM apps were just starting out, and ChatGPT was the biggest thing? Even though LLMs couldn’t directly operate systems or data, people already started to "vibe something". They could give helpful advice, like step-by-step guides to get things done.

With the above ideas and observations in mind, we decided to introduce an Agent to our system. Aside from thinking about how humans can understand and use the tool, let’s focus on how Agent (AI) can understand and access it, because the investment has a very high return which brings these benefits:

Lower the entry barrier: Our users can ask basic questions, know almost nothing to get started, because the Agent can give them instructions via Q&A iteration.
Streamline complex tasks: Instead of clicking through endless menus or filling out lengthy forms, users can simply tell the Agent what they need. Think of it as having a super-smart assistant that anticipates your needs.
Reduce development time: By letting the Agent handle some of the user interactions, we can reduce the amount of custom UI development needed. Plus, less hand-holding for every single new feature is a major win! (Busy platform team 🥵)
Enhance user experience: A conversational interface can make using our tools feel more intuitive and less like wrestling with a computer. It’s like teaching our tools to speak our language, not the other way around.
Increase flexibility: The Agent can adapt to different user needs and preferences on the fly, making our platform more versatile and user-friendly. We can even add new functionalities without needing to redesign the whole interface! (Who doesn’t love skipping a redesign meeting or two?)

After intensive development and workshops, our team brought the very first version of this Agentic UX into our platform. Here’s a quick peek into our progress!

From Rough Draft to Reality: Building an AI Assistant

From a quick glance, yeah, it might look like just another AI chat tool, and honestly, at first, that’s kinda what it was! It allows users to attach sources, check references, and even has a "thinking process" we designed ourselves. Pretty standard AI fare.

But here’s the catch – for us, just "pretty standard" wasn’t gonna cut it. We needed super high accuracy. If this thing messed up, it wouldn’t just be a minor glitch, it could be a major incident generator. Imagine accidentally sending out the wrong promotion to thousands of users! Not exactly a "oops, my bad" situation.

So, we went deep into the rabbit hole. Massive prompt engineering? Check. Implemented more guardrails than a bowling alley? Double check. Created new designs to connect the Agent seamlessly into our existing systems and UI? You betcha. It was like trying to teach a brilliant, but slightly chaotic, intern how to perfectly follow a super complicated set of instructions.

Achieving production-level quality with AI is far more than just "magic"; it demands significant engineering effort to ensure accuracy and reliability. It’s not enough for AI to simply talk; it must consistently say the right things to be a dependable tool.

Conclusion: Just the Tip of the AI-berg

So, this is definitely not the end of the story. In fact, it’s really just the beginning.

The whole AI world is changing everything around us, and we’re basically just learning how to swim in this new AI tide. We’re adapting, experimenting, and maybe splashing around a bit too much. But hey, you gotta start somewhere!

What we’ve really done here is open the door. We’ve built a foundation to bring the future of AI’s superpowers to our platform. We’re talking about AI that not only talks but understands, anticipates, and makes our tools smarter than we ever imagined. This first version of the Agent? It’s just the first step on a much longer, much more exciting journey. And we can’t wait to see where it takes us (and our users!).

Tomorrow’s article will be by @toshinao from the Mercoin Ops Team.. Look forward to it!

Removing GitHub PATs and Private Keys From Google Cloud: Extending Token Server to Google Cloud

Tue, 27 May 2025 15:25:14 GMT

At Mercari, we have been working on reducing the number of long-lived credentials that could have a significant impact on our systems if leaked and abused. In order to achieve this we have implemented multiple systems that issue short-lived credentials. The Platform Security Team has extended an internally operated service called Token Server, which generates GitHub credentials, so that automated services running on Google Cloud can switch to short-lived credentials for accessing GitHub.

This article introduces the technologies, challenges, and solutions behind extending Token Server and migrating workloads on Google Cloud to use short-lived credentials.

Overview

Mercari primarily uses GitHub as its development platform, and we develop and operate many services that automate GitHub-related tasks.
These services typically access GitHub with a Personal Access Token (PAT) or a GitHub App private key, which can have no expiration or very long expiration periods. If such credentials are leaked (for example, through a supply chain attack), they can be misused for a long time. Also, once these long-lived credentials are created, it can be unclear which service uses which credential, and there is rarely a review of their granted permissions.

To resolve these problems, we extended an existing Token Server service (which already issues short-lived GitHub credentials inside Mercari) so that any service running on Google Cloud could also access GitHub without using long-lived credentials. This change provides the following benefits:

Reduction of the number of long-lived credentials
Reduction in the number of both PATs and GitHub App private keys (often managed in non-transparent ways)
Simplified process for identifying which service uses which credential and for periodically reviewing permissions, by consolidating credential assignment and required privileges into one place

Moreover, we developed a Go library that allows existing services to migrate to Token Server with minimal changes, enabling quick adoption while avoiding major rewrites.

Token Server

At Mercari, GitHub is used in many different ways. In particular, for GitHub automation, it is common to implement changes in one repository and apply them to another repository automatically.
With GitHub Actions (our standard CI platform), there is no default way to handle automation across multiple repositories. Usually, you must store a PAT or GitHub App private key in Repository Secrets and generate tokens using, for example, the create-github-app-token action.
However, these methods require long-lived credentials (PAT or a GitHub App private key).

To address this, Mercari has been running a Token Server service that issues an Installation Access Token with certain permissions, by verifying an OIDC token that GitHub provides inside GitHub Actions workflows.

Installation Access Tokens are part of GitHub App functionality. They can be restricted to a subset of permissions (for example, read permission for contents, write permission for pull requests) and limited to certain repositories. They expire after one hour and can also be revoked via the GitHub API before they expire. This means you can provide credentials limited by the principle of least privilege, granting only the necessary scope, access range, and lifespan.

The architecture of Token Server for GitHub

Token Server creates Installation Access Tokens from a pre-configured GitHub App, based on permissions for each repository and branch, and provides these tokens to GitHub Actions jobs in that repository. To identify which repository and branch to associate, the Token Server uses the OIDC token available inside the GitHub Actions job. The job obtains the OIDC token, sends it to the Token Server, which verifies the token, looks up the permissions set for that repository and branch, and then creates and issues an Installation Access Token.
Installation Access Tokens issued by Token Server are used for a wide range of activities, such as multi-repository automation (adding commits, automatically creating issues, pulling requests) and downloading private libraries during builds.

(Note) In April 2024, Chainguard released Octo STS. Its core principle is similar to Token Server. However, Token Server provides more unified permission management and also integrates with Google Cloud workloads and GitHub App load balancing. This makes it well suited for enterprise environments.

Token Server’s Extension to Google Cloud

At Mercari, many services run on Google Cloud. This includes not only customer-facing microservices but also internal services for automation. These services accessed GitHub using PATs or GitHub App private keys.

Each Google Cloud resource has a Service Account that can be granted privileges to operate other resources. When a Google Cloud resource has the roles/iam.serviceAccountTokenCreator permission, it can obtain an OIDC token signed by Google via an API. We decided to extend the Token Server to verify these Google-signed OIDC tokens just like we do with GitHub’s OIDC tokens, so we can issue an Installation Access Token with predefined permissions.

The architecture of Token Server for Google Cloud

With this approach, a service running on a given Google Cloud resource can send an OIDC token to the Token Server, receive an Installation Access Token, and then use it to access GitHub – eliminating the need for previously stored PATs or GitHub App private keys in Google Cloud.

Applying Token Server to Workloads on Google Cloud

By extending Token Server, services on Google Cloud can now switch their GitHub access credentials to a short-lived token.

It is relatively easy to apply these new features to newly created services on Google Cloud. However, for many existing services that have already been using a PAT or GitHub App private key, implementing the process of requesting an Installation Access Token from Token Server and then using it can be difficult.

Moreover, GitHub Apps have a rate limit on API usage: 15,000 requests per hour per GitHub App on GitHub Enterprise Cloud. Exceeding this rate limit causes API requests to fail. Because Token Server can serve multiple Google Cloud workloads and multiple repos, it is critical to reduce the total number of requests.

It is also important to note that the rate limit covers not only the number of token issuance requests to the Token Server but also all API traffic made using each issued Installation Access Token. Instead of requesting a new Installation Access Token for every single GitHub API call, the approach is to reuse the same token within its one-hour validity period, thus reducing the overall requests.

Migration from PAT to Token Server in GitHub client initialization

To avoid major rewrites in existing services and to automatically obtain and reuse an Installation Access Token within its validity period, we developed a library. Because Mercari mostly uses Go, we built this library on top of the google/go-github library, which is widely used in Go-based GitHub automation. If an existing service already uses go-github, the service can migrate to Token Server simply by configuring the Service Account and replacing the library.

Library Structure for Token Server

When you initialize the go-github library, you can specify any http.Client. The http.Client uses a custom RoundTripper implementation that can modify the request before it is sent. We leverage this RoundTrip method to check if the cached Installation Access Token is still valid. If it has expired, we request a new Installation Access Token from Token Server; otherwise, we reuse the existing one.

The process of Token Server library

With this design, existing services only need to change a single line of code to migrate to Token Server (if they already use go-github).

GitHub App Load Balancing

As mentioned before, each GitHub App has a rate limit of 15,000 requests per hour. Token Server will potentially handle a large number of API requests from multiple Google Cloud workloads and multiple GitHub repositories. We also expect an increase in automated services over time, so we must be prepared for traffic that could exceed these limits.

To handle this, we considered creating multiple GitHub Apps and distributing requests among them to avoid hitting a single GitHub App’s rate limit. However, if a load balancer randomly distributes requests to multiple Token Server pods, each loaded with a different GitHub App, a single user might receive tokens from more than one GitHub App.

This becomes an issue for a service that writes commit statuses. In GitHub, you can record statuses (error, failure, pending, success) for a single commit. These statuses are tracked per GitHub App. If multiple GitHub Apps post statuses for the same commit, the statuses become mixed. In a workflow where the first step might post a failure status and a later step posts a success status, these statuses need to come from the same GitHub App to overwrite properly. Otherwise, you could end up with a failure status from GitHub App 1 and a success status from GitHub App 2, which could block merges if branch protection requires all statuses to pass.

Writing statuses with multiple GitHub Apps

If the first failure status comes from GitHub App 1, a subsequent success status from GitHub App 2 cannot overwrite it. This results in mixed commit statuses that can prevent merging.

To solve this, we assign the same GitHub App consistently for each target. One Token Server pod can load multiple GitHub Apps, then choose which GitHub App to use based on the repository and branch name (on GitHub) or the Service Account (on Google Cloud).

The assignment process of GitHub Apps

By mapping GitHub Apps according to repository, branch name, or Service Account, we ensure that the same GitHub App is always used for the same repository, branch, or Service Account.

Summary

By extending Token Server to Google Cloud, more services can use short-lived credentials for GitHub, reducing the need for long-lived credentials. We also developed a library that lets existing services migrate to Token Server with minimal changes. Through these efforts, we solved issues discovered during real-world operations, supporting more secure and efficient GitHub automation at Mercari.

The Mercari Security Team will continue working on replacing long-lived credentials with short-lived ones.

For information on careers in the Security Team, please see Mercari Careers.

When Caching Hides the Truth: A VPC Service Controls & Artifact Registry Tale

Fri, 23 May 2025 15:00:31 GMT

Hello, I am South from the Mercari Platform Security team.

To mitigate potential impacts of Docker Hub rate limits and improve supply chain security, Mercari has undertaken a project to launch an in-house Docker registry and migrate our production infrastructure over to pull from the registry. This project mainly involved Google Artifact Registry and VPC Service Controls.

This post will cover the reason behind the project, the solution we chose, an outage that was caused during the rollout and the lessons learned.

Impetus: The Docker Rate Limit Announcement

This project began in response to the announcement of new Docker Hub rate limits. The announcement, giving about one week’s notice, set an initial effective date of March 1, 2025.

We promptly started investigating systems in our company infrastructure that pull from Docker unauthenticated and drafted plans to ensure that these systems pull from Docker with credentials. While Mercari primarily builds and uses in-house containers, a small number were pulled from official upstream sources, including some base images from Docker Hub.

Later, we noticed that the new restriction has been delayed by a month to April 1, 2025, and we continued our planning.

Deciding on a Solution: the Registry Part

We evaluated several potential solutions. Google hosts a Docker Hub mirror at mirror.gcr.io, which caches "frequently-accessed public Docker Hub images". For images not cached by mirror.gcr.io, Google recommends using an Artifact Registry remote repository. (While our tests indicated direct pulls of uncached images via mirror.gcr.io might sometimes work, we followed the official guidance.) An Artifact Registry remote repository allows configuring Docker Hub credentials, ensuring reliable upstream image fetching without hitting rate limits. Alternatively, we could have configured Docker Hub credentials individually wherever image pulls occur, but this approach was deemed too labor-intensive and error-prone.

Considering critical use cases like our production cluster and CI/CD infrastructure, alongside the need for developers to pull images, we opted for the Artifact Registry route. Having chosen Artifact Registry, we started considering how to handle authentication between the image puller and the remote repository to prevent running a public Docker registry and potentially incurring substantial costs.

Setting the Stage: What are VPC Service Controls?

Before we dive into our solution for the authentication, let’s set the stage with a quick primer on VPC Service Controls.

VPC Service Controls (VPC-SC) is a Google Cloud feature for defining a service perimeter around specified resources. It controls both ingress (access from outside the perimeter to resources inside) and egress (access from inside the perimeter to resources outside). While ‘VPC’ is in the name, these perimeters can secure access to resources based on the project they reside in, which was key for our Artifact Registry setup.

Note: VPC-SC is tightly related with Access Context Manager (ACM): all VPC-SC APIs are under the accesscontextmanager.googleapis.com domain, and many VPC-SC resources (for example, ingress rules) can refer to ACM resources (for example, access levels). In this article, we will use VPC-SC to refer to both VPC-SC and ACM, since it is not likely that we will use VPC-SC alone.

A service perimeter in VPC-SC typically contains Google Cloud projects and can restrict access to specific services within those projects. Conceptually, VPC-SC establishes this security perimeter around the specified resources. By default, this perimeter blocks network communication crossing its boundary.

To allow approved communication, administrators configure ingress and egress rules. These rules define specific exceptions, permitting authorized traffic through the perimeter under defined conditions. Crucially, ingress and egress refer to where the principal accessing the resource and the resource being accessed are located with respect to the access boundary, not necessarily the direction of data flow. For example, we need to configure an ingress rule to allow a user outside of the boundary to download a sensitive file from a bucket inside of the access boundary, despite the sensitive data flowing outwards.

Rather than detailing all rule configurations, let’s consider a concrete example relevant to our use case. Suppose we want to allow users from a specific corporate IP range to access images from an Artifact Registry instance within a specific project. To achieve this:

An access level must be created defining the specific IP range.
An ingress rule must be configured for the perimeter, specifying this access level, the intended users (or service accounts), the target project, and the artifactregistry.googleapis.com service.

This configuration permits users from the specified IP range to access the registry, while access from other locations remains blocked by the perimeter.

Deciding on a Solution: the Authentication Part

Both IAM permissions and VPC-SC can manage access to Artifact Registry. However, certain internal workloads required the ability to pull images from specific IP ranges without easily configurable authentication mechanisms. Standard IAM role bindings alone could not satisfy this requirement.

IAM supports various principal identifiers. The allUsers identifier grants access to any principal, including unauthenticated users, whereas allAuthenticatedUsers restricts access to authenticated Google accounts. A notable consequence of using either principal identifier is the disabling of data access audit logs for the registry.

Given that this registry mirrors only public images, confidentiality was not a requirement. This allowed us to deviate from our usual identity-first approach and instead use network controls (IP filtering) to efficiently prevent costly, unauthorized external access. Implementing IP-based restrictions without altering numerous client applications necessitated using the allUsers binding on the Artifact Registry repository, thereby shifting the burden of access control entirely to the VPC-SC perimeter’s IP filtering rules.

This approach, using allUsers on the registry and relying on the VPC-SC perimeter for actual IP-based filtering, was necessary to meet our requirement of allowing pulls from specific internal systems without embedding authentication credentials into each one. While configuring the IAM policy and referencing the relevant IAM documentation, the side-effect of allUsers inhibiting data access logs was not apparent, as this detail resides mainly in separate audit logging documentation. The significance of this logging behavior emerged during the subsequent incident response.

Rolling Out: Dry-Running & Going Live

To validate our configuration safely, we utilized VPC-SC’s valuable dry-run mode. This feature logs potential policy violations that would occur if the policy were active, without actually blocking traffic, sending details of these potential denials to the audit logs. In Terraform, dry-run mode can be enabled using the use_explicit_dry_run_spec flag and specifying the intended policy within the spec block.

After enabling dry-run mode for several days, we analyzed the audit logs to identify any legitimate traffic that would be inadvertently blocked and prepared the necessary additional ingress rules. The audit log provides details on the request, source identity and IP address, and destination service, enabling us to refine the policy.

Following the dry-run period and necessary rule adjustments, we enabled the VPC-SC restrictions in active mode. In Terraform, this involved disabling use_explicit_dry_run_spec and moving the policy definition from the spec block (for dry-run configuration) to the status block (for active configuration). Initially, registry operations continued without apparent issues.

When Things Go Wrong: The Incident Unfolds

Several days after enablement, a planned update was required for the registry’s Docker Hub credentials. Originally, the registry pulled upstream images anonymously, but to avoid potential rate limits, we configured it through Terraform (this part will come into play later) to use an API token stored in Secret Manager.

This update unexpectedly led to image pull failures for end-users. We began an investigation into the cause. The investigation faced challenges: data access logs were unavailable (a consequence of the allUsers setting), standard VPC-SC violation logs were not being generated for this failure mode, and the client error message provided only a generic "caller does not have permission". The recently enabled VPC-SC perimeter was identified as a likely factor. To restore service quickly while continuing the investigation, we decided to temporarily revert the VPC-SC enablement, temporarily resolving the issue after 68 minutes.

Digging Deeper: The Incident Investigation Process

Once the revert was complete and image pulls were functional again, we continued the investigation.

The investigation revealed that the root cause actually predated the credential switch. A missing VPC-SC config had been present since enablement, but its effect was masked by Artifact Registry’s image caching mechanism. When we switched the credentials using Terraform, the Artifact Registry repository resource was unnecessarily recreated due to a Terraform provider bug, clearing the cache. While we noted the planned recreation of the repository, we didn’t anticipate issues, assuming images could simply be re-fetched from the upstream source. However, this cache clearing exposed the underlying VPC-SC configuration gap. At this point, Artifact Registry needed to pull images directly from Docker Hub but was unable to do so.

The core technical issue was that Artifact Registry required network egress to reach Docker Hub, and this path was blocked by the VPC-SC perimeter. Allowing this traffic requires a dedicated VPC-SC config (google_artifact_registry_vpcsc_config in Terraform) specifically for Artifact Registry remote repositories. Crucially, this isn’t managed via standard egress rules; it requires a dedicated configuration designed solely to allow these repositories to bypass the perimeter for upstream fetches. No egress rules, even ones that permit all egress, would allow this traffic. This crucial configuration was missing in our initial setup.

Regarding the absence of VPC-SC violation logs for this failure, Google Cloud Support confirmed this is the expected behavior for this specific Artifact Registry egress scenario.

Furthermore, we discovered a limitation in the dry-run mode’s coverage: it did not generate violation logs for this specific scenario (blocked upstream pulls by a remote repository due to missing google_artifact_registry_vpcsc_config), even though the active policy would block the traffic. We only knew the cause of the problem because Google Cloud support was able to point out the issue with the information we had provided. Fortunately, despite anticipating no disruption, our deployment plan included performing the rollout during hours when the team was available for immediate incident response, which proved essential.

After creating the necessary VPC-SC config for the remote repository, we re-enabled the restriction. This time, image pulls functioned correctly, even with an empty cache.

Learning from Experience: Retrospective Findings

Our post-incident review confirmed the missing VPC-SC config as the direct cause. The review also highlighted related areas for improvement,

Lack of visibility into the status: early in the incident response, the absence of relevant logs made determining the cause of the failure difficult. This required us to rely primarily on available Artifact Registry metrics and deductive reasoning to identify the root cause of the image pull failures.
- Remediation: We now understand that using the allUsers binding inhibits data access audit log generation for certain events. This finding has been shared within our team and with other relevant teams. Going forward, we will explicitly consider this logging limitation as a known trade-off when evaluating the use of allUsers.
Lack of a comparable staging environment: while we had a testing environment and performed tests before applying the same changes to the production environment, the testing environment is not similar enough to the production environment, notably that it lacks the same downstream pullers to allow us to detect problems that did not pop up during testing but happened during the incident.
- Remediation: even though we do not have plans to make changes to the registry yet, we have started creating a staging environment parallel to the production environment, with consumers of the registry that pull images from the staging environment to ensure that we will be able to catch as many problems as possible during the next change.
Insufficient breakglass access: during the incident response, we had tried to speed up the changes by bypassing CI and making changes with our breakglass access. While we were able to approve the breakglass request quickly, we discovered that the breakglass access role does not grant sufficient access to perform the changes.
- Remediation: we made a change to the breakglass access role after the incident response. In addition, we are planning additional incident response training and tabletop exercises to catch similar issues.

We have since taken action to address some identified hazards and continue to work on others.

Final Thoughts: On VPC-SC and Third-Party Dependencies

While powerful, the complexity of VPC Service Controls necessitates careful configuration and deep understanding, sometimes making alternative solutions preferable. If implementing VPC-SC, a thorough grasp of its mechanisms combined with rigorous testing (including dry runs) is essential for a successful and secure deployment.

In addition, learning from this experience, we recognize the risks associated with free third-party services, particularly how their terms can change unexpectedly. Consequently, we are adopting a more cautious stance moving forward. We will prioritize the stability and predictability offered by in-house solutions or paid services with explicit agreements, thereby minimizing our reliance on free external services wherever possible.

From DNS Failures to Resilience: How NodeLocal DNSCache Saved the Day

Mon, 19 May 2025 03:51:51 GMT

About us

I am Sanu Satyadarshi, part of the Platform Engineering division at Mercari, Inc. Platform Engineering provides a cost-effective, safe, and easy-to-use multi-cloud infrastructure service for all engineering teams to make and scale bets.

Summary

This article discusses the DNS-related challenges encountered at Mercari on our Kubernetes clusters and the significant improvements achieved by implementing Node-Local DNS Cache. By optimizing DNS traffic and reducing errors, we enhanced system reliability and scalability, preventing production outages caused by DNS failures.

DNS queries before and after the rollout of Node-Local DNS Cache.

Key Takeaways

Reduced DNS calls to kube-dns by 10x, decreasing network overhead and inter-service communication costs.
Lowered DNS query rates by 93% for services on the cluster.
Achieved a 10x-100x reduction in DNS-level errors, improving system resilience.
Eliminated the "failed to refresh DNS cache" errors, mitigating a frequent source of incidents.

DNS on Kubernetes: The Elephant in the Room

Domain Name System, more commonly known as DNS an extremely critical component in the internet infrastructure. This is the tech that allows your web browser to find the actual IP address of a website when you type example.com in your browser. DNS in itself is a highly complex topic, and understanding it requires a book(or two) on its own.

Like any network infrastructure, Kubernetes depends on DNS to resolve service names like [service name].[namespace].svc.cluster.local and other names to IPs and allows communications among services and the external world.
From the role of DNS in Kubernetes, you can imagine that any DNS failure or degradation can quickly escalate to increased latency, network congestion, and even complete outages.

On Kubernetes, DNS is installed as a kube-dns deployment running on the kube-system namespace. Specifically at Mercari, it comes pre-installed with our managed GKE clusters for service discovery and name resolution across the clusters.
kube-dns on Kubernetes allows multiple configurations using the configmap that can be used to change various parameters like ndots, etc.

As kube-dns is responsible for resolving all the service queries to IP addresses, scaling the kube-dns pods in response to the number of pods, etc., is the most logical step.
Fortunately, Kubernetes provides kube-dns autoscaling by default to deal with such high-traffic clusters like ours.

Our DNS Challenges

At Mercari, our Kubernetes clusters process extremely high RPS during peak hours, where we started seeing the limitations of kube-dns.

High DNS query rates were overwhelming the kube-dns service.
Frequent DNS-level errors, including NXDOMAIN and truncated responses.
Recurring "failed to refresh DNS cache" errors were causing cache misses.

The final nail in the coffin was a Sev1 incident where multiple services started to fail DNS resolution, leading to timeouts and, eventually, a production outage due to the cascading nature of microservices.

Node-Local DNS Cache: Our Saviour

Previously, for any DNS queries, all the services relied on a few kube-dns pods to resolve the domain names like [service name].[namespace].svc.cluster.local to the IP address of the Service(aka Endpoints).

This setup used to overwhelm the kube-dns pods and caused issues that we talked about in the previous section.

Node-Local-DNS Cache provides a radically different approach to handling DNS queries. Instead of relying on the few kube-dns pods, it uses the tried and tested concept of caching at the Kubernetes node level. This allows all the pods on a particular node to use the DNS cache on that node before reaching out to the kube-dns pods.

Source: kubernetes.io

This provides multiple benefits:

Localized DNS resolution, reducing inter-node traffic.
High scalability of the cluster during peak business hours.
Reduction of load on kube-dns, thus providing resiliency against kube-dns failures

Implementation

Once we identified the solution, we started planning the rollout strategy for node-local-dns-cache across all our environments.

To do a gradual rollout and reduce the blast radius, we deployed the NodeLocal DNSCache on our Laboratory GKE Cluster(which is only used by the Platform Teams for internal testing) with a specific nodeAffinity. This allowed us to safely measure the impact of NodeLocal DNSCache without impacting all the workloads.

Based on our learnings, we decided to gradually roll out NodeLocal DNSCache across all our Dev and Prod environments by adding labels on the node pools to allow NodeLocal DNSCache pods to be deployed.

Impact and Results

The results were unbelievable.

10x reduction in DNS calls to kube-dns.
A 10x to 100x reduction in DNS-level errors depending on the class of error (e.g., 10x for nxdomain, 100x for truncated)
100% elimination of "failed to refresh DNS cache" errors, which were responsible for many production incidents.
Significant improvement in cluster scalability and network efficiency.

DNS Error count before and after the rollout

DNS Query rate before and after the rollout

Conclusion

Implementing Node-Local DNS Cache addressed our DNS challenges, resulting in a 10x reduction in DNS traffic, fewer errors, and enhanced system reliability. These improvements underscore the importance of optimizing DNS in Kubernetes clusters, especially for high-traffic environments like ours. By sharing our experience, we hope to guide others in enhancing their DNS operations and achieving similar results.

I would like to thank Yusaku Hatanaka (hatappi) and Tarun Duhan for their valuable inputs and contributions during the implementation.

Upgrading ECK Operator: A Side-by-Side Kubernetes Operator Upgrade Approach

Mon, 28 Apr 2025 12:00:23 GMT

Greetings, I’m Abhishek Munagekar from the Search Infrastructure Team at Mercari. Our team manages several Elasticsearch clusters deployed on Kubernetes, forming a crucial part of our search infrastructure. We rely on the Elastic Cloud on Kubernetes (ECK) Operator to orchestrate these clusters, all housed within a dedicated namespace maintained by our team.

To leverage the advancements in recently released ECK operator versions, we embarked on an upgrade project. Operator upgrades are inherently complex and risky, often involving significant changes that can affect system stability.

In this article, I’ll delve into the challenges we encountered and the strategies we employed to manage operator upgrades for stateful workloads like Elasticsearch. Additionally, I’ll detail how we modified the ECK operator to facilitate a more resilient side-by-side upgrade process.

Minimizing Risk in a Critical Infrastructure

At Mercari, our Elasticsearch infrastructure is integral to multiple business units, notably powering the marketplace search functionality. Any disruption or downtime to this infrastructure carries the potential for significant financial repercussions. Therefore, our primary objective during ECK operator upgrades is to mitigate risk to the absolute minimum. This necessitates a cautious and strategic approach, favoring gradual rollouts over abrupt big-bang deployments, employing side-by-side upgrades instead of in-place replacements, and ensuring robust disaster recovery plans.

We utilize a suite of safety nets and backup mechanisms, including Elasticsearch snapshots, real-time write request backups, standby cluster preparations, and rigorous testing across multiple environments. While the details of these mechanisms are extensive, they fall beyond the scope of this particular article.

In-place Upgrade Mechanism used by the Native ECK Operator

Typically, Kubernetes operators, including the native ECK operator, perform in-place upgrades, where an existing component is directly replaced with a newer version. In contrast, a side-by-side upgrade involves running two versions of the same component concurrently. Here’s a comparative overview:

Feature	In-place Upgrade	Side-by-side Upgrade
Downtime	Possible	Minimized
Rollback	More Difficult	Feasible
Resource Usage	Lower	Higher (Double)
Complexity	Lower	Higher
Examples	OS upgrades	Database Upgrades

In-place upgrades carry inherent risks, particularly with stateful workloads like Elasticsearch. If issues arise, rollback is complex and time-consuming, leading to prolonged recovery periods. This is in contrast to stateless workloads, where recovery is generally faster and less risky.

Limiting Standard ECK Upgrades

A standard ECK operator upgrade triggers a rolling restart of Elasticsearch nodes across all clusters simultaneously. This all-at-once approach is unacceptable for our high-stakes production environment, where a more gradual rollout is essential. The ECK operator offers an annotation, eck.k8s.elastic.co/managed=false, to temporarily unmanage Elasticsearch clusters, allowing for one-by-one upgrades.

However, this solution conflicts with our infrastructure’s CPU-based autoscaling mechanism. Our system monitors data nodeset CPU usage and scales Elasticsearch by modifying the manifest, with the ECK operator provisioning the necessary nodes. Disabling the operator’s management effectively halts our autoscaling (detailed in this blog article).

One workaround would be to manually scale workloads to maximum capacity, apply the unmanaged annotation, and then proceed with a serial upgrade process, by removing the unmanaged annotation one at a time.

Following is a flowchart for the proposed plan.

But this was rejected for the following reasons:

Costly: Disables crucial autoscaling features.
Inflexible: Prevents scaling during unexpected traffic surges.
Restrictive: Blocks any configuration changes to Elasticsearch during the upgrade.

Our Solution: A Custom Side-by-Side Upgrade Strategy

To circumvent these limitations, we chose to implement a custom side-by-side upgrade approach that mimics the granular control of eck.k8s.elastic.co/managed=false but is tied to the operator’s version.

Introducing Operator Version Labeling

We introduced a new label:

eaas.search.mercari.in/desired-controller-version = x.y.z

This label is applied to all Elasticsearch clusters, initially set to the current (older) operator version. We then modified the ECK operator’s logic(referencing this GitHub link) to recognize this label and control cluster management accordingly.

Modifying the Controller for Dual Version Support

Both the existing (older) and the new ECK operator versions were modified to support this label. Functionally, we adapted the IsUnmanaged function and the main controller loop to:

Check for the eaas.search.mercari.in/desired-controller-version label.
Skip reconciliation if the label is missing or if the label’s version does not match the operator’s build version.

Here’s the relevant code snippet:

const desiredECKControllerVersionLabel = "eaas.search.mercari.in/desired-controller-version"

func IsUnmanaged(ctx context.Context, object metav1.Object) bool {
    managed, exists := object.GetAnnotations()[ManagedAnnotation]
    if exists && managed == "false" {
        return true
    }

    desiredVersion, exists := object.GetLabels()[desiredECKControllerVersionLabel]

    if !exists {
        ulog.FromContext(ctx).Info(fmt.Sprintf("Object doesn't have %s label. Skipping reconciliation", desiredECKControllerVersionLabel), "namespace", object.GetNamespace(), "name", object.GetName())
        return true
    }

    if desiredVersion != about.GetBuildInfo().Version {
        ulog.FromContext(ctx).Info(
            fmt.Sprintf("Object is not the target of this controller by %s label. Skipping reconciliation", desiredECKControllerVersionLabel),
            "desired_version", desiredVersion,
            "operator_version", about.GetBuildInfo().Version,
            "namespace", object.GetNamespace(),
            "name", object.GetName(),
        )
        return true
    }

    paused, exists := object.GetAnnotations()[LegacyPauseAnnoation]
    if exists {
        ulog.FromContext(ctx).Info(fmt.Sprintf("%s is deprecated, please use %s", LegacyPauseAnnoation, ManagedAnnotation), "namespace", object.GetNamespace(), "name", object.GetName())
    }
    return exists && paused == "true"
}

Handling Custom Resource Definitions (CRDs)

The ECK operator defines a custom resource of Kind: Elasticsearch. This is a cluster-scoped resource, not a namespaced resource, so we cannot define two distinct versions of the CRD concurrently within the same cluster.

In this scenario, we rely on the backward compatibility of the CRD definition. It’s crucial to note that while CRDs are expected to be backward compatible, they may not be forward compatible. Backward compatibility ensures that older operator versions can work with newer CRD definitions. However, forward compatibility, which would mean newer operators can seamlessly work with older CRD definitions, is not guaranteed.

This implies that the latest version of the CRD must be deployed to the cluster when running two different versions of the ECK operator side-by-side. Failure to do so could lead to issues where the newer operator version cannot interpret newer CRD fields or configurations, resulting in deployment or operational errors. Therefore, before initiating an upgrade, ensuring the newest CRD version is applied is a critical prerequisite.

Handling Validating Webhook

ECK also defines a validating webhook, which validates the Elasticsearch manifests before they are applied to the cluster. When running two versions of the ECK operator concurrently, it is crucial to ensure that each operator version only validates the Elasticsearch clusters for which its desired-controller-version matches.

The default webhook configuration, without any restrictions, would mean that an Elasticsearch manifest could be validated by both versions of the operator. This poses a significant risk because newer operator versions might introduce new features or modifications to the validation logic. These changes could render validation performed by an older operator version incompatible or incorrect for the expectations of the newer operator, or vice versa. This discrepancy could potentially lead to deployment failures, configuration errors, or unexpected behavior.

Instead of modifying the controller logic itself, a simple object selector was added to the webhook configuration.

 objectSelector:
    matchLabels:
      eaas.search.mercari.in/controller-version: x.y.z

This objectSelector with matchLabels ensures that each ECK operator version only validates Elasticsearch manifests that have the corresponding desired-controller-version. By isolating the validation process based on the operator version, we prevent potential conflicts and ensure that manifests are only validated by the operator version that is expected to manage them.

Leader Election for High Availability in ECK Operator Upgrades

The ECK operator employs leader election to ensure high availability. Multiple instances of the operator can run concurrently, but only one acts as the active leader responsible for processing changes. This leader election mechanism relies on Kubernetes leases, specifically by acquiring a Kubernetes lease.

In a standard, in-place upgrade scenario, the ECK operator uses a constant Kubernetes lease named elastic-operator-leader. Regardless of the operator version, they all contend for this same lease. When an in-place upgrade occurs, the new operator version simply replaces the old and takes over this existing lease.
The following diagram illustrates the leader election process during a standard in-place upgrade:

However, the default lease strategy presents a challenge for our side-by-side upgrade approach. Since both the older and newer ECK operator versions would try to acquire the same elastic-operator-leader lease, it would result in contention and only one version of the operator could run at a given time. To facilitate our dual-version scenario, we needed a way to separate the leader election for each version.

To address this, we modified the ECK operator’s leader election logic to create distinct Kubernetes leases based on the operator’s version. This ensures that each operator version has its own separate leader election process, allowing them to run in high availability side-by-side without conflict.

We made changes to the LeaderElectionID in the ECK operator code. This ID now includes the operator’s version:

func GetLeaderElectionLeaseName() string {
    buildInfo := about.GetBuildInfo()
    k8sVersion := strings.ReplaceAll(buildInfo.Version, ".", "-")
    leasePrefix := "elastic-operator-leader-v"
    leaseName := fmt.Sprintf("%s%s", leasePrefix, k8sVersion)
    return leaseName
 }

LeaderElectionID: GetLeaderElectionLeaseName()

In essence, this change transforms the default elastic-operator-leader lease into version-specific leases, such as elastic-operator-leader-v2-16-1 for version 2.16.1. With these versioned leases, each ECK operator instance will only participate in leader election with instances of the same version. The following diagram shows the leader election process with our side-by-side upgrade:

Testing Our Approach Thoroughly

The Search Infrastructure Team at Mercari leverages three distinct environments to ensure the stability and safety of our infrastructure changes:

Laboratory Environment: This environment serves as a dedicated playground, allowing the infrastructure team to rigorously test changes without impacting the development environment. It’s our sandbox for experimentation and initial validation.
Development Environment: This environment mirrors the production setup to a significant degree and is primarily used for Quality Assurance (QA) testing and the development of new features. This is where we validate changes under conditions closely resembling those in production.
Production Environment: This is the live environment serving real user traffic, demanding the highest level of stability and reliability.

Before any production deployment, changes are meticulously tested in both the laboratory and development environments. We conduct comprehensive testing to ensure both the older and newer versions of the ECK operator can coexist without conflicts. This includes verifying the labeling system, controller logic modifications, CRD handling, and validating webhook changes. We also perform thorough rollback tests to guarantee that we can quickly revert to the previous state if issues arise. This rigorous testing across multiple environments is crucial to minimizing risk in our high-stakes production environment.

Rollout to Production: A Phased and Monitored Process

Our production rollout follows a phased and closely monitored approach to minimize risk. This involves:

Preparation: Verify CRDs and webhook configurations are compatible with the new operator version.
Labeling: Tag all Elasticsearch clusters with eaas.search.mercari.in/controller-version set to the current operator version for tracking.
Dual Deployment: Deploy both old and new ECK operators concurrently.
Gradual Rollout: Upgrade clusters incrementally by updating their labels to point to the new operator version (eaas.search.mercari.in/controller-version=<new_version>) cluster-by-cluster.
Continuous Monitoring: Track key metrics like error rates, system stability, and resource usage during each upgrade.
Validation & Rollback: After each cluster upgrade, validate success or rollback by reverting labels and configurations if needed.
Completion: Upgrade remaining clusters, validate, and then remove the older operator version.

The following diagram illustrates the workflow that we follow.

Conclusion

In summary, upgrading critical systems like the ECK operator needs careful planning and testing. Mercari’s specific needs led us to create a unique side-by-side upgrade strategy. By carefully changing the operator and using a step-by-step release, we successfully reduced risks and kept our search system running smoothly.

It’s often hard to perfectly copy real-world workloads in testing environments. This can lead to bugs slipping through. This challenge highlights a limitation of the standard approach, as standard operator upgrades are usually tested in development before going to production all at once.

While Kubernetes applications use methods like gradual releases and canary deployments, operator upgrades often use an all-at-once method. We found this wasn’t ideal for our critical search infrastructure.

With our successful ECK operator upgrade using the side-by-side approach, we plan to use this strategy for other critical operator upgrades in our production system. We hope our approach helps other teams manage Kubernetes operators, especially those which handle stateful workloads.

gcp-sa-key-checker: A recon tool for GCP Service Account Keys

Fri, 25 Apr 2025 16:02:45 GMT

Today Mercari is open sourcing gcp-sa-key-checker, a recon tool for keys attached to GCP Service Accounts that does not require any permissions. In this post I’ll provide some background about GCP Service Account security, provide the motivation for the project, and then describe the tool and some findings.

Background: GCP Service Account Keys

GCP Service Accounts (SA) are the primary Non-human Identity (NHI) principal type in the GCP IAM model. They are normally identified by an ’email’ like my-service-account@project-id.iam.gserviceaccount.com and can be granted permissions to cloud resources the same as users or other principals.

Service Accounts each have a collection of RSA Service Account Keys attached to them, some of which are always Google Managed and some which can be User-Managed. The public portion of these keys is shared as a JSON Web Key Set (JWKS), so that JWTs assigned with them can be verified as legitimate. These JWTs can then be used to authenticate as the service account to Google or any other service that trusts the JWKS.

Note: It might be surprising to some to learn that, because Google Managed service account keys are always 2048-bit and the public portions are published to the internet (not to mention that internal service account emails are easily guessable) almost all workloads on GCP very directly rely on the security of 2048-bit RSA keys.

The private portion of Google Managed keys is always held by Google and can never be accessed by users, however Google does provide oracle access to these keys through the signBlob, signJwt and generateIdToken methods which are authorized via regular IAM bindings.

In contrast, User Managed keys exist outside of Google Cloud and their security is entirely managed by the user. The key material for these can be either generated by Google and downloaded ("Google Provided") or generated locally and the public portion uploaded ("User Provided"). Google strongly recommends against using User Managed service account keys:

You should choose a more secure alternative to service account keys whenever possible. If you must authenticate with a service account key, you are responsible for the security of the private key and for other operations described by best practices for managing service account keys.

At Mercari, in line with the GCP best practices, we’ve used Org Policy Constraints to prevent users from creating or upload user-managed SA keys in the general case. My team has granted a small number of exceptions for external tools that only support SA keys, such as GitHub Audit Logs streaming to GCS (ticket) or GCP’s own CCAI Service (ticket), strictly under the condition that we have an open tracking issue/feature request with upstream to support keyless authentication.

What about third party service accounts?

After being hit hard by the codecov compromise in 2021, Mercari has heavily invested in removing long-term credentials from our own environment, including for GCP. This includes projects such as cleaning up usage of GCP SA Keys, and reducing usage of long-lived GitHub PATs (although unfortunately the gh auth token still lives forever).

However, in addition to our own SAs, we also have various external SAs that are connected to our GCP environment. These accounts are operated by various SaaS vendors for the tools we use for functions such as Observability, FinOps and CSPM, but also Google itself. We were wondering, could we also check if these service accounts have user managed keys attached to them?

A careful reading of the documentation revealed that in addition to the JWKS endpoint for each SA, there is also an x509 public key endpoint that Google warns against disclosing private information in:

For uploaded service account keys, the X.509 certificate provided by the public endpoint is the same certificate as the one you uploaded. If the certificate you uploaded contained any optional attributes (such as address or location information embedded in the common name), then this information also becomes publicly accessible. A bad actor might use this information to learn more about your environment.

Downloading the 509 certificates for several test accounts, we found that there were clear differences between the certificates attached to Google Managed and User Managed keys, particularly in the validity period. So, we decided to build a tool for automatically checking accounts based on these heuristics.

The tool: gcp-sa-key-checker

You can find the tool now on GitHub at github.com/mercari/gcp-sa-key-checker, and the README contains details on running it. For supplied Service Accounts, it will guess if each key was generated by Google or the User, and which manages the key material. We’ve run this internally against >20k SAs, and found no issues with the heuristics.

We used Wiz to find all external service accounts referenced from our cloud footprint, then used the tool to scan them. We found that some of our vendors seem to not be following the best practices for User Managed SA keys. In particular, it seems that some are using long-lived, downloaded (instead of uploaded) keys to access our environment, which is something that we’ve disallowed internally.

For example, we identified that one external partner’s SA had 6 total Google-provided User-managed keys without expiry that have access to one part of our environment. Checking the audit logs, it is clear this principal is only used from GCP IP addresses which suggests that service account keys should not be necessary. We plan to follow up with this and other vendors in private to inquire about their key management practices.

In the future, we hope that this recon method can be incorporated into other tools to continue to promote keyless authentication methods for GCP. If you have any questions or feedback about the tool, please direct it to the GitHub page!

My Two-Month Internship Working on Mercari Hallo

Wed, 09 Apr 2025 11:11:28 GMT

Hello, my name is @masa, and I am a first-year graduate student at Kyushu University.
I did a two-month frontend engineer internship at Mercari, working on Mercari Hallo, at the end of 2024.

Left to right: Me (@masa) and my mentor @d–chan

In this post, I’ll talk about my area of interest, strategy for integration testing, and what I learned at Mercari during my internship.

Why I decided to do an internship to work on Mercari Hallo

My main goal of doing an internship working on Mercari Hallo was to experience service development for a large-scale service, particularly a consumer-facing service. Mercari Hallo is one of Mercari’s services and was released less than a year ago, so it is still a relatively new product. Working on Mercari Hallo provided the perfect opportunity for me to learn about practical development processes in a field that demands speed and quality.

Another reason why I chose Mercari was to gain first-hand experience of Mercari’s workstyle and culture for a better understanding of how such a company operates.

Initiatives for integration testing

During my time as an intern, I worked on different tasks of different sizes. One project I was particularly invested in was integration testing for business-facing UI screens. When I joined, the team had already determined which technology to use and had finished creating the development environment under the guidance of our tech lead @ryotah, and was just about to start working on improving test coverage.

At the time, integration tests for Mercari Hallo were performed one page at a time based on specifications, using frontend testing methods previously used at Merpay. I worked on the following two improvements to this process:

Avoiding bloated code
Optimizing validation testing

Avoiding bloated code

Writing tests according to the specifications ensures consistent test granularity and policy throughout the team. However, sticking too closely to the specifications means that, for example, the same code is written to validate the same form components on different screens, which tends to make the code bloated.

To solve this problem, we considered the following three approaches:

Write tests for shared components
Advantages: The problem of redundant code can be solved. The same test can be used for shared components, which means that there is no need to write the same validation logic over and over again.
Disadvantages: Taking this approach would deviate slightly from the "test in a way that’s close to how the application actually works" policy for integration testing. There is also the concern that different people will write tests in different ways if complex portions are treated as components.
Writing one test for all screens
Advantages: With both of these intermediate approaches, developers write code that is faithful to testing based on the specifications, which were written with how users will actually use the page in mind. Because of this, it is easier to notice slightly different use cases and bugs.
Disadvantages: Writing a large amount of similar test logic makes editing that logic a big job and maintaining the code difficult.
Write tests for shared components on one representative screen
Advantages: With the two intermediate approaches mentioned above, it’s possible to maintain basic functionality while keeping test redundancy to a minimum.
Disadvantages: This approach is not completely comprehensive, so it may be necessary to write additional tests for other pages.

In the end, we decided to write tests for shared components on one representative screen and write additional tests only when there is page-specific logic. Considering team resources and development speed at the time, we determined that this was the most realistic and flexible approach.

Optimizing validation testing

Unit testing covers standard validation using the form library (react-hook-form), so for integration testing, we focused on any parts that are difficult to validate with unit testing.
For instance, schema testing using react-hook-form alone cannot cover the logic that displays a modal when there is a submission error, as shown below.

const onSubmit = (value) => {
  // if input field contains an error
  if (value.name !== 'hoge') { 
    setShowModal(true)
  }
 // data transmission, etc.
}

A part like this can be validated with an integration test using Playwright.

// Example of integration test using Playwright
test('display modal if input field contains an error', async({page}) => {
  // omitted
  // ...
  await page.getByLabel('name').fill('foo');
  await page.getByRole('button', {name: 'send'}).click();
  await expect(
      page.getByRole('dialog', { name: 'include keyword in name' }).toBeVisible();
});

I made sure to balance the cost and ROI of writing test code and write meaningful test code that doesn’t create any technical debt.

Also, to increase transparency and efficiency of the development process, I created a Slack channel for integration testing. I created this channel because there wasn’t really anywhere to ask for advice about technical issues in the frontend domain, and because there were few opportunities to communicate with engineers in other teams. In this channel, we could share any questions we had or specific problems we faced during implementation, which led to a shared sense of problem awareness across the team and helped us find better solutions.

Other activities and experiences

During my time as an intern, I also participated in an ideathon aimed at improving the efficiency of work using generative AI.

In the allotted 90 minutes, I worked in a team to come up with ideas and even create a prototype. While the schedule was very tight, it was a very exciting and fun experience.

When choosing which idea to present, we focused on whether other people experienced the problem we were trying to solve and whether we could achieve a result in a short amount of time. In the end, we went with an idea called “C’mon, Calendar!” which aimed to streamline scheduling on Google Calendar based on participants’ availability and the type of events people want to add.

Everyone on my team was so talented, and I struggled to see how I could contribute at first. Focusing on my strengths, I decided to create the workflow and handle implementation. We wanted to get the prototype to a point where we could use Zapier to retrieve calendar information, but unfortunately we ran out of time.

I’m pleased to announce that my team won the ideathon! 🎉
(Thank you to all my team members! 🙇‍♂️)

Difficulty communicating in English

When I interviewed for the internship, I was told that the team I would be joining did not use much English so I didn’t need to have strong English skills. However, between that interview and me joining the company, some team members changed, and I had to participate in a weekly all-English frontend engineer meeting from my very first week! I was worried about being able to communicate in English, and I really struggled when I had to facilitate the meeting in English. I used cheat sheets and other tools to help me get through.

Mercari has a lot of non-Japanese employees, so I had plenty of opportunities to use English when attending events at the office. Also, pull request reviews are made in English, so I got to experience working in an English-based environment.

At first I was taken aback by how often I had to speak English, but being in that environment really motivated me to study more. Working somewhere that improved both my technical skills and global communication skills really helped me grow as an engineer.

To conclude

Through my Mercari Hallo internship, I was able to gain a lot of valuable experience in the field of large-scale service development. Implementing integration tests in particular gave me great insight into how to write high-quality and effective test code and the importance of team communication.

I feel that the knowledge and experience I gained over those two months will serve me well in my future studies and career. Lastly, I’d like to thank my mentor @d–chan and everyone who welcomed me to the company.

Tackling Knowledge Management

Mon, 10 Mar 2025 11:00:20 GMT

Introduction

Hello! I’m @raven from Mercari’s Engineering Office.
This article is an English translation of a Japanese article I wrote for Day 14 of the Mercari Advent Calendar 2024 series.

Mercari’s Engineering Office is a team that works to solve problems and challenges faced by engineers across Mercari Group. Improving knowledge management for our engineering organizations is also part of our job.

When I joined Mercari in April 2024, I felt that it was hard to find knowledge. I had to ask coworkers where I could find the information I needed; I didn’t know where knowledge owned by other teams was stored, nor how to go about looking for it.

Right around the same time, we carried out an annual survey targeting engineers across Mercari Group, and internal knowledge ranked as the area with the highest level of dissatisfaction. Just as I was pretending to be surprised, I got a request to be part of a project to improve knowledge management—talk about luck!

What you’ll find in this post

We haven’t reached the finish line with this project yet, but I’d like to share what we’ve done so far to increase satisfaction with knowledge management among engineers. Specifically, I’ll talk about the following two points:

What approaches we took to solving the problems faced by engineers
How we drove the project across Mercari Group

I hope this post is useful to anyone out there facing similar knowledge-related problems in their own organization.

Dissatisfaction with knowledge management among engineers

Improving knowledge management is much easier said than done. It requires asking engineers to make changes to the culture of documentation they’ve cultivated over the years. This is a difficult process even within just one organization; the scope of my team had just expanded from a single product division to all engineering organizations in Mercari’s Japan Region, including our India office. That made this knowledge management project a great initiative for us, perfect for our mission of solving problems and challenges faced by engineers across Mercari Group.

We began by analyzing engineers’ responses to the survey that showed dissatisfaction with knowledge management. The major sources of dissatisfaction seemed to be the following points:

Knowledge is scattered across multiple platforms, making it hard to search for or find what you’re looking for
There are many different knowledge platforms, but each organization has their own rules for building knowledge, so the knowledge isn’t centralized or organized
There isn’t a standard format for documentation, so even the same type of document may have different content and be written in a different style depending on the organization that owns it
No one is actively maintaining knowledge, so there are many cases of outdated or redundant knowledge
There are no training programs or guidelines regarding knowledge management
Some documents are in English and some are in Japanese; the language barrier makes it hard to share information

If you’ve made it this far, you’re probably nodding in agreement with at least some of these points.
The loss caused by not managing knowledge appropriately is greater than any of us could imagine—for both the company and for engineers.

We started this project envisioning a world where our engineers could share and find information stress-free, across organizations and languages.

How to approach each of these problems

After looking through the comments from engineers, we determined that we needed to solve the following problems:

Knowledge is scattered across multiple platforms
Knowledge isn’t organized because there are no consistent rules
Because of the first two points, it’s hard to search for or find what you’re looking for
Information isn’t shared widely enough because of the language barrier
Documentation isn’t standardized
Knowledge is not appropriately maintained
There are no guidelines or training programs about knowledge

In the next section, I’ll share our approaches to tackling each of these problems.

Problem: Knowledge is scattered across multiple platforms

We mainly use three tools to create documentation:

Confluence
Google Docs / Slide
GitHub (knowledge collected and published as webpages)

When it came time to select a platform to manage our knowledge, there were many different opinions about using our existing assets. For example, one dramatic approach suggested was to use Confluence as our only platform. But when we compared these products, we determined that each of them had different advantages.

Product	Advantages
Confluence	Page creation is intuitive; knowledge and knowledge domains are easy to manage
GitHub	Offers features such as version management, reviews, and approvals
Google Workspace	Seamlessly integrates with various collaboration tools

After a lot of discussion, we decided that our policy would be to use Confluence as our main knowledge platform, and use other platforms as necessary to supplement the features that Confluence is missing.

Problem: Knowledge isn’t organized because there are no consistent rules

We decided on a flexible design for our knowledge platform in order to leverage the advantages of each tool, but allowing the use of multiple tools runs the risk of not actually solving the problem of information being scattered across different tools. To prevent this, we used organizational structure information to automatically create Confluence pages for each organization’s knowledge domain dedicated to storing the knowledge of all teams in that organization. We had each team fill out a standardized template with information such as their communication channels, GitHub repositories, and design specs, to assemble team information that is worth sharing internally as knowledge on Confluence in a consistent format regardless of organization.

We chose to organize knowledge in this way mainly because, given the current organizational structure and chain of command, categorizing the information by team would make it easier to implement governance and drive projects forward. We also considered categorizing the information by product or by tech domain, but we thought that as the first step toward improving knowledge management, the team-based approach was the best way to clarify who is responsible for what knowledge as we move ahead with this project.

Organizing information on the same team level across all of Mercari Group also had the important purpose of enabling engineers to understand the information and knowledge held by other organizations more easily. Personally, I feel that this was like drawing a map by hand of an uncharted world—it’s rough and not very detailed, but it still gives us a broad view of the different organizations across the company.

Problem: It’s hard to search for or find what you’re looking for

By linking information on Confluence, we made it a little easier to follow a link trail to each organization’s knowledge. However, just placing links doesn’t make it dramatically easier to search for knowledge.

You may have noticed the arrow from Confluence to LLM + RAG in the knowledge platform diagram. From the beginning of the project, we’ve been working with our Large Language Model (LLM) Team to see if it’s possible to use a retrieval-augmented generation (RAG) solution for information to enable engineers to search engineering knowledge on Confluence. The LLM Team had already imported the main sources of engineering knowledge on GitHub into a RAG, so we decided to do the same for information on Confluence that would be useful to engineers and provide that knowledge using internal LLM systems.

Problem: Information isn’t shared widely enough because of the language barrier

Engineers who can’t understand Japanese well won’t read documentation in Japanese. Engineers who can’t understand English well won’t read documentation in English. It may seem obvious, but breaking down the language barrier is crucial to enabling engineers to seamlessly share knowledge.
That said, we don’t have the resources to write all documents in both Japanese and English, and Confluence’s translation plugin cost scales based on use, so using Confluence as our main knowledge platform comes with a potential impact on cost.

Thankfully, we already have LLM and RAG solutions, so we decided to use them to solve the language issue for knowledge that should be shared in both Japanese and English. Using our LLM system, engineers can ask questions in Japanese and receive answers in Japanese, even if the content comes from documentation written in English. We expect this to facilitate seamless sharing of knowledge regardless of differences in language and contribute to engineers discovering knowledge they may not have had the chance to find before.

Problem: Documentation isn’t standardized

Before this project, most documentation was written using templates that each organization had defined as their own standard. For more complex cases, some organizations even had multiple different templates.
Using one standardized template across organizations ensures that each document provides information in the same level of detail and enables anyone to create documentation with just the right amount of information. It also reduces the stress readers may face when they try to find and understand the information they’re looking for. Therefore, we decided to first recommend the use of standardized templates for the types of documentation most frequently created by engineers.

Problem: Knowledge is not appropriately maintained

In order to ensure that knowledge is kept up to date, we enhanced the “health check” tool we use for documentation on Confluence. This tool enables us to monitor and visualize the freshness of information, the usage status of standardized templates, and other data. We periodically request that engineers run these checks as a way to manage knowledge maintenance.

Problem: There are no guidelines or training programs about knowledge

To help engineers understand our knowledge management initiatives, we created guidelines on Confluence regarding choosing documentation tools and using standardized documentation templates. We plan to expand these guidelines going forward.

That said, we know that not all engineers will read through the guidelines and immediately change their habits to follow them. We used our internal e-learning system to create a training course on our fundamental approach to knowledge management and the content of the guidelines, and made it a mandatory course for engineers in order to promote understanding of the guidelines and a change in mindset regarding knowledge management.

In addition to this training, we are also taking other actions to ensure that engineers understand how important knowledge management is, like sharing information at company-wide meetings for engineers and holding periodic open-door sessions.

Driving a Mercari Group-wide project

Just deciding how to approach the problems faced by engineers isn’t enough—you can have the best idea in the world, but it’s meaningless if you can’t commit to and follow through with the plan.

In this section, I’ll go over some points we were particularly careful about when driving this project across Mercari Group.

Project design
Visualization
Forming a knowledge management committee
Following up with information owners (IOs)
Announcements and awareness-raising activities

Project design

Throughout this knowledge management improvement project, we carefully considered the outline of our initiatives, the schedule, detailed tasks, risk assessment, the plan for spreading awareness of appropriate knowledge management, training, monitoring plans, and more.
We also created a project management Confluence page with this information and worked to actively publish information to increase recognition of our initiatives among both project members and other employees.

Visualization

We visualized our plans and initiatives using diagrams to ensure that they would be easy to understand for project stakeholders and other employees. In meetings, using visual images of our initiatives helped participants understand the content more accurately and quickly, enabling seamless understanding across the group.

Forming a knowledge management committee

Even within the same company, different organizations have different cultures and habits surrounding documentation.
In order to drive this project forward across Mercari Group, we first selected information owners (IOs) to act as representatives of knowledge management within each organization and formed a knowledge management committee. There were about 20 IOs in the committee. We worked together with these IOs to consider how to share documentation between organizations, the best policies for documentation across the group, guidelines, training content, and more. When collecting knowledge owned by each team, each IO asked the managers in their organization to update the information. They also encouraged the members of their organization to take the training course. Thanks to this committee, we were able to work together to improve knowledge management.

Following up with IOs

In an ideal world, IOs would be able to focus all of their time and energy on the knowledge management project, but in reality, they’re busy with their own work. Not all IOs can participate in committee meetings, so we assigned each IO a representative project member in the Knowledge Management Team and held individual one-on-ones to follow up with IOs and minimize any information gaps.

Announcements and awareness-raising activities

Just releasing guidelines or training programs is pointless if engineers don’t actually read them. We do make announcements on communication channels, of course, but announcements aren’t enough to ensure that all engineers know about the guidelines and programs and take the appropriate action. We worked with IOs to apply knowledge management methods in their organizations and actively raised awareness of the importance of knowledge management among engineers through company-wide meetings for engineers and open-door events.

Conclusions

In this post, I wrote about our initiatives to improve knowledge management in our engineering organizations and key points for driving the project across Mercari Group.

Knowledge management initiatives don’t stop when the project is over; we still have to periodically reflect user feedback in our guidelines and training programs, expand and encourage use of standardized templates, import knowledge into LLMs, and more. We will continue to strive for further enhancements to a sustainable knowledge management culture for engineering at Mercari.

Once we have established a knowledge foundation for engineering, we’d like to expand our knowledge management initiatives to product and business areas as well to cover the entire company.

If you made it this far, I hope our experience provided some valuable insights.
Thank you!

Redesigning the International C2C Shopping Experience for Mercari Taiwan

Mon, 03 Mar 2025 13:32:03 GMT

We are excited to announce the launch of Mercari in Taiwan, which allows Taiwanese customers to purchase items directly from our extensive Japanese marketplace. In this article, I will delve into the value proposition behind the new user experience for Mercari Taiwan, which aims to create a seamless shopping journey for international customers.

A Marketplace of Global Opportunities

As the largest C2C marketplace in Japan, Mercari offers a diverse selection of items that attract both domestic and international customers. Japanese pre-loved items are highly valued, and unique offerings are available — particularly in anime, comics, and gaming categories.

However, international customers faced a significant barrier: they couldn’t directly purchase items or create an account on Mercari Japan. Instead, international customers used proxy services, which served as intermediaries to facilitate purchases on their behalf. These proxy services maintained accounts on Mercari Japan and provided functionalities, including:

Placing orders on Mercari Japan
Receiving items at their warehouses
Conducting item checks
Finalizing orders with sellers
Shipping items internationally

Proxy services are essential in creating a seamless experience for Japanese sellers by managing communication and shipping logistics with international buyers. Consequently, these proxy services have become the sole avenue for international buyers to tap into Mercari’s extensive inventory, yet the buying process proved to be complicated and cumbersome.

Navigating the Proxy Experience: A Complicated Journey

Using the proxy service involved a multi-step process that often overwhelmed customers:

Search for an item on Mercari
Navigate to the proxy website to locate the same item
Check out on the proxy site and make the first payment
Wait for the item to arrive at the proxy service’s warehouse in Japan
Receive an email prompting a revisit to the proxy site
Choose a shipping method and make the second payment
Finally, receive the item

This complex process presented several UX challenges:

Customers struggled to understand how to use the proxy service.
The purchasing journey was lengthy and required significant time and effort.
Customers had to constantly switch between Mercari and proxy websites.

Consequently, this intricate experience primarily attracted heavy customers while deterring light customers. Ultimately, it hindered our ability to scale the business effectively.

New User Experiences: Streamlining the Cross-border Purchase Journey

To tackle these challenges, we focused on designing a new experience that empowers international customers to purchase items directly from Mercari. Key enhancements in our approach include:

Enabling customers to complete all transactions on the Mercari website
Shortening the purchase process by implementing a one-time payment system
Improving the overall shopping experience through refreshed checkout screens and clear post-transaction communication

In this new user experience, customers now enjoy a more streamlined process:

1. Search and find an item on Mercari:
Benefit from a personalized and consistent browsing experience that enhances discoverability.

2. Checkout with a single payment:
Navigate through clear instructions and intuitive navigation, making the checkout process straightforward.

3. Receive the item:
No additional actions are required after checkout; simply wait for the item to arrive at home.

This streamlined experience enables customers to bypass the complexities of the proxy service, enjoying a straightforward and efficient purchasing process. The purchased item is sent to the warehouse in Japan first and is then shipped overseas, just as before.

The new UX solutions encourage light customers to engage in international shopping while keeping dedicated ones interested with an improved experience, thus facilitating scalable business growth.

A Step into the Future

With the launch of this new user experience in Taiwan, Mercari is poised to redefine the international C2C marketplace experience for both current and future customers. We are committed to continuous exploration, updates, and expansions of our user experience, ensuring each customer enjoys a seamless and rewarding shopping journey.

We look forward to sharing our progress as we move ahead in this exciting new chapter for Mercari Taiwan. Thank you for your support as we set out to enhance your shopping experience!

From Local to Global: How Mercari Expanded to Taiwan in just 8 Months

Fri, 28 Feb 2025 10:00:04 GMT

Exactly 6 months ago, on 29th August 2024, we rolled out Mercari to Taiwan for the first time.

A portion of Slack message from the main project manager in charge of InHouse project (cropped for succinctness).

The Spark of a Global Dream

Imagine a project that starts with three simple words: "Make Mercari Global." Sounds easy, right? As the Frontend (FE) Person In Charge (PIC) of project InHouse, I can tell you it was anything but simple. I want to show the behind the scenes of how the Crossborder (XB) team transformed an ambiguous vision into a concrete reality, bringing Mercari’s marketplace magic to Taiwan.

I will be talking about the project management side, frontend side, and the aftermath 6 months later. Please skip to the part you are interested in 🙂

Project Management: Turning Vagueness into Vision

When leadership drops a goal like "Make Mercari Global" on your desk, you could panic. Or you could do what we did: break it down, strategize, and execute with precision.

Why Taiwan?

Currently international users can purchase Mercari items through third party services. From these services we know which countries and regions have demand for Mercari items. Taiwan (台湾) sits high on the second place.

Ranking of countries and regions by amount of purchase from XB. Taiwan is in second place. Image taken from XB transaction trends.

If we further break down the data by popular categories, we see that Taiwan (台湾) ranks highly on all of them.

Ranking of countries and regions by amount of transactions. Taiwan ranks 4th, 2nd, 2nd, 3rd, 2nd for badges, kpop CDs, idol goods, acrylic stamps, and figurines respectively. Image taken from XB transaction trends.

It’s also worth noting that for the initial release we planned to only ship one item at a time. And Taiwan ranks even higher for this metric. Do note that we will soon allow users to order and ship multiple items at the same time, so look forward to that!

Taiwan was chosen over China due to complex licensing requirements, strict data laws, and product certification needs. The sheer size and competition within China makes it a fairly difficult country to release first to.

Taiwan was chosen over the USA for 2 main reasons. Firstly, Mercari already has a presence in the USA through Mercari US. Secondly, Taiwan is geographically much closer to Japan; meaning that we can minimize the shipping cost.

Managing Time: The 8-Month Marathon

Planning an 8-months project that touches every single codebase and screen is like conducting an orchestra where every musician is playing a different genre. Our approach? Sync, sync, and sync even more…
People seem to hate meetings, but I think there’s a time for it. Projects with specs that changes daily and confirmation that requires long contexts is one of them. And man did we have a lot of meetings.

At the peak of it we had

(30m) Daily standup team meetings where the team can sync on daily changes
(1hr) Weekly Product/Engineering meetings where each team updates their statuses
(1hr) Weekly section meetings. For example, I participated in engineering meetings where we discussed technical blockers and approaches. I would also join meetings where we check the schedules and ensure we are still on track to deliver the project.
(30m) 1on1 with each stakeholder. I mainly have these with PICs from each department
(1hr) Some sections of the project are quite large and also have their own kickoff and weekly sync meetings. For example, the authentication (handled by Daniel) section had their own weekly sync meetings.

So each week, almost a day’s worth of hours are spent on meetings. This is a lot, but also essential in keeping all the context in sync. Meetings were also one of the ways to highlight critical issues and resolve them quickly.

Having meetings is not an excuse to not properly document decisions. We still ensured every decision is documented and not just left on Slack. Thank you to Aymeric (EM) and Nick (Pdm) for organizing and running many of these meetings.

Managing People: Split loads and break silos

There are 2 forces pulling each other

You want to enable people to have deep work. To do this you need to minimize meetings and contexts required for a task.
You also want knowledge to not be siloed. To do this you need to maximize syncs and contexts for a task.

See how each part contradicts each other? Our approach was simple, have 1 PIC for each department (PM, FE, BE, Design, etc) who will have the full context of everything. And then delegate various tasks to other team members.
Simplified frontend team report line diagram.

FE for example split our works by functions. Some examples are

Purchase flow (Wills)
Internationalization (Drew)
Authentication (Daniel)
MyPage (Gary)
etc

Members of each section can then ignore the other and focus on delivering their work. As PIC, I need to keep the context of everything. This means any question from another team or department can be asked to me. This speeds up communication across the team. I also document each important decision so when needed, other members can refer to it.

Each team member can dive deep into each functionality and contact the external team for advice and guidance. This allowed fast work by each member without bypassing the codeowners of respective screens or functionality.

Frontend: The Center of Chaos

Frontend plays a central role in this project. The Frontend team ties BE, Design, PM, legal, and other teams together. As such, keeping our heads clear and being on top of all the specs was a must.

The Repo Dilemma: New or Existing?

One of our first big decisions: create a new repository or modify existing ones? We went with modifying our existing one as various infrastructures were already set up. For example, release flow, on-call, and staging was already set up.

Internationalization (I18n): More Than Just Translation

I18n wasn’t just about switching languages. It was about creating a seamless experience that felt native to Taiwanese users. Note that up until now Mercari was only available in Japanese.
We established rigorous standards. Some of these might be obvious, but writing them down and enforcing them was important in order have good standards

Standardized URL structures for static pages. This is especially important when navigating from pages that are managed on different repositories.
- Use case sensitive BCP 47 standard right after domain name (e.g. jp.mercari.com/zh-TW)
Consistent file system organization
- This follows the above where we will have parent folder named as the locale (e.g. html/en/cookie_policy)
UI, flow, and fallback
- If you have multiple language options, always show a language picker from all pages
- Store selected language locally and sync when users are signed-in
- When a language is only partially available, default to en or ja depending on the language
Clear decision-making flows for localization
- Start with Figma
- Export strings to a CMS
- FE names the keys
- Internal or external team translates the strings
- FE pull latest changes and commits to the codebase

Controlling UI

FE worked closely with Masa (Designer) and other designers to keep the UI and UX consistent between countries. If you are interested in our UI/UX decisions for Taiwan release, please check Redesigning the International C2C Shopping Experience for Mercari Taiwan article written by the design PIC.

Without a doubt, there are sections where the UI must be different. To achieve this, we have a few methods we can use.

By feature flag

This is our current system for doing A/B testing. If you are not familiar with A/B testing, this article by nngroup is a great starting point.

We split the UI depending on whether a feature flag is true or false. For example, we have the XBT-2974_int_cvs_pickup feature flag. The values are set using an internal system, but all it does is randomly distribute values to existing users. If a user receives a false value then they will not see this new feature. If a user receives a true value then they will see the new feature.

# feature flag definition file
const featureFlags = [
    ...
    'XBT-2974_int_cvs_pickup',
    ...
];

# file where we want to make the split
export const Component = (props: Props) => {
    ...
    const { getFlag } = useFeatureFlag();
    ...

    return (
        ...
        {getFlag('XBT-2974_int_cvs_pickup') ? <NewComponent> : <OldComponent>}
        ...
    );
};

By country

We can also control the UI based on the user country. When signed in we retrieve the user’s country from the DB. When signed out we retrieve the user’s country from our CDN (which determines it using ip address).

# file where we want to make the split
export const Header = (props: Props) => {
    ...
    const isInternationalUser = useIsInternationalUser();
    ...

    return (
        ...
        {isInternationalUser && <LanguagePickerButton />}
        ...
    );
};

By sign in state of user

This is especially useful for pages that should only be accessible by signed in users.

# file where we want to make the split
export const UserPreferencePage = () => {
    ...
    const signIn = useIsSignIn();

    useEffect(() => { 
        if (!signIn) { 
            loginRedirect(true);
        } 
    }, [loginRedirect, signIn]); 

    if (!signIn) { 
        return null; 
    }

    return (
        ...
    );
};

Access Control List (ACL)

To have a more robust access control per user type, user country, and feature, we developed an access control list. This is more complicated and also involves BE. Shoutout to Gary for implementing this.

If you have never heard of Access Control List, then the Access Control chapter from Operating Systems: Three Easy Pieces (OSTEP) is a great starting point.

# permissions
export const permissions = {
    ...
    # Japanese users need to be multi authenticated to use following Shops feature. Since Taiwan has not been set, Taiwanese users can't use this feature.
    SHOPS_FOLLOW: [ 
        createPermission(
            AccountCountryCode.JP,  AuthenticationContextClassReference.MultiFactor
        ), 
    ],
    ...
}

# file where we want to make the split
export const ShopsFollowLink = () => {
    ...
    const { isFeatureAvailable} = useACL();
    ...

    return (
        ...
        {isFeatureAvailable(FeatureId.SHOPS_FOLLOW) && <FollowShopsButton>}
        ...
    );
};

We employed each method depending on the feature and design. As a rule of thumb, we start with a feature flag for simple A/B features. When it’s more complicated, we use the more complicated ACL. And finally use country/regions or sign in state when that is the only variable we are interested in.

Aftermath

Marketing plans

Successfully launching to a new country/region is a technical achievement, but it is just the beginning. Next we need to make sure the time and effort we invested are paid off. Now, I’m no expert at marketing and business development, so please understand that the following section has my very personal takes.

Mercari is a household name in Japan, but not in Taiwan.That being said, Mercari is quite known for some categories of items (read Anime, Manga, Game, and Idol products). To play into our strength we set up various marketing campaigns targeting these markets.

W11

Singles’ Day lands on 11/11 every year. The date was chosen for how it resembles single people. It is especially huge in Asia since Alibaba started offering huge discounts back in 2009. As this will be Mercari’s first big event in Taiwan, we went with a bang. Setting up offline booths and huge discounts for 2.5 weeks leading in 11/11 and on the day itself (press release in Japanese).

Online campaign page for W11 (campaign page in Taiwanese).

W11 offline event entrance.

W11 offline event 2000s drama themed room.

The W11 event was very successful. Mercari’s name was successfully spread out by influencers taking photos in the offline booths. Taiwanese people are now more than ever aware of our service.

Huge discounts also nudge users for registration and first purchase (2 of the hardest blockers for marketplace).

Christmas

Who doesn’t love Christmas? Mercari definitely does! Hoping to entice users looking for Christmas presents we set up discounts throughout the Christmas period.

Online campaign page for Christmas (campaign page in Taiwanese).

Although it wasn’t as big as the W11 event, the Christmas event was still very successful. We exceeded most of our targets, including many new users and first purchases.

Other marketing events

With the marketing team (shoutout to Moty and Angie) in Mercari working hard, the InHouse project gained over 50,000 users in just the first month! Mercari will continue to hold offline events and campaigns to promote the service so spread the word to your Taiwanese friends as their next purchase might be highly discounted from Mercari!

What’s Next?

The global expansion train has left the station, and we’re just getting started. We are building more features for our Taiwanese users to use our service even easier and cheaper. We will also continue to hold fun events to help promote the brand.
At the same time Mercari will continue to expand to other countries in the following months. Keep your eyes open as Mercari might be available in your country soon!

Thank you for reading! <3

LLM x SRE: Mercari’s Next-gen Incident Handling Buddy

Thu, 06 Feb 2025 14:41:47 GMT

I’m Tianchen Wang (@Amadeus), a new graduate engineer of the Platform Enabler team at Mercari, Inc. In this blog, I will share our new progress with creating Mercari’s Next-gen incident handling buddy by utilizing the Large Language Model (LLM).

In today’s fast-paced technological landscape, maintaining a robust on-call operation is crucial to ensuring seamless service continuity. While incidents are inevitable, the ability to swiftly respond and resolve them is essential for assuring users a safe, stable, and reliable experience. This is a shared goal among all Site Reliability Engineers (SREs) and employees at Mercari.

This article introduces IBIS (Incident Buddy & Insight System), an on-call buddy developed by the Platform Enabler Team leveraging generative AI. IBIS is designed to assist Mercari engineers in rapidly resolving incidents, thus reducing the Mean Time to Recovery (MTTR), and reducing on-call handling costs for companies and engineers.

Challenges and Motivation

At Mercari, ensuring that users can safely and securely use our product is a paramount goal and vision shared by all employees. To this end, we have established an on-call team of multiple divisions working together. Each week, on-call members receive numerous alerts, a significant number of which escalate into incidents that impact users. These incidents result in poor user experiences and an increase in Mean Time to Recovery (MTTR), which negatively affects Mercari’s business and product offerings.

Additionally, on-call members must devote considerable time to handling these incidents, indirectly reducing the time available for developing new features and impacting our ability to achieve business objectives.

As a result, reducing MTTR during incidents and mitigating the burden on on-call members have become critical challenges for the Platform team. With the advent of Large Language Models (LLMs), automating incident handling through their integration has emerged as a potential solution.

Deep dive: Architecture

Let’s take a closer look at the architecture of our incident handling system “IBIS”.

Fig 1. Architecture of IBIS

From a high-level perspective, we extract past incident retrospective report information from our incident management tool, Blameless. These reports include data such as temporary measures, root causes, and damages caused by the failures. This data undergoes cleansing, translation, and summarization processes. Subsequently, we utilize OpenAI’s embedding model to create vectors from these data sources.

When users pose questions to our Slack bot using natural language, these queries are also converted into vectors. The conversation component then searches the vector database for embeddings related to the question, and formulates a response to the user by organizing the relevant language constructs.

Let’s break down the entire architecture into two main components for detailed explanation: Data processing and Conversation.

Data processing

Below is the way how IBIS pre-process incident data.

Fig 2. Data process progress of IBIS

Export data

Our incident management tool Blameless includes the process details of each incident, chat logs from incident Slack channels, retrospective reflections, and follow-up actions, among other vital pieces of information. We utilize Google Cloud Scheduler to regularly export the latest incident reports from Blameless’s external API into a Google Cloud Storage bucket. This process is designed to align with serverless principles and is executed within Google Cloud Run Jobs.

Data cleansing

We cannot indiscriminately send data obtained from Blameless into a Large Language Model (LLM). This is not only because the data contains numerous templates, which can significantly affect the precision of our vector searches (Cosine Similarity), but also because it includes a substantial amount of Personally Identifiable Information (PII). To mitigate the risk of potential information leakage and enhance the accuracy of the generated results, data cleansing is a necessary process.

To remove templates from the data, we leverage the fact that the data is in Markdown format and use the Markdown Splitter function provided by LangChain to extract relevant sections. As for PII, since it has multiple types, we opted to employ the SpaCy NLP model for tokenization and remove potentially existing PII based on word types.

The data cleansing component runs on Google Cloud Run Functions. From this stage onwards, we use Google Cloud Workflow to manage the entire system. When a new file is added to the Google Cloud Storage Bucket, Eventarc automatically triggers a new workflow. This workflow uses HTTP to initiate the data cleansing Cloud Run Function and, upon completion, proceeds to the next stage in the process, as shown in Figure 2. Introducing Cloud Workflow facilitates easier code maintenance throughout the ETL process.

Translating, summarizing & embedding

The cleansed data is then forwarded to the next stage of the process. Thanks to data cleansing, we can now confidently utilize the LLM model to process data smarter. Since both Japanese and English are used for writing incident reports at Mercari, translating these reports into English is a critical step for enhancing search accuracy. We utilize GPT-4o-based LangChain to handle the translation step. Moreover, since many reports are lengthy, summarizing the content is also crucial for improving vector search precision. GPT-4o assists us in summarizing the data as well. Finally, the translated and summarized clean data undergoes embedding and is stored in our Vector Database.

The translation, summarization, and embedding processes run on Google Cloud Run Jobs. Once data cleansing is complete, the Cloud Workflow automatically triggers a Cloud Run Job. As depicted in Figure 2, the embedded data is stored in our BigQuery Table using the BigQuery vector store package provided by LangChain.

Conversation

The Slack-based conversation feature is a core function of IBIS. In our design, users can directly engage with IBIS through natural language questions by mentioning the bot in Slack. To achieve this functionality, we need a server that continuously listens for requests from Slack and can generate responses based on our Vector Database.

Fig 3. Conversation System for IBIS

As illustrated in Figure 3, this server is built on Google Cloud Run Service. It retrieves relevant information from BigQuery, which acts as our Vector DB, and then sends the data to an LLM model to generate responses.

In addition to handling queries, the conversation component also supports other functionalities, such as short-term memory, enhancing the interaction experience.

Short-term memory

Considering that an engineer’s understanding of an incident evolves over time, incorporating memory functionality within the same thread is vital for enhancing IBIS’s ability to resolve incidents and provide recommendations. As shown in Figure 4, we utilize LangChain’s memory feature to store both the user’s queries and the LLM’s responses from the same channel. If additional queries are posed in the same channel, the previous conversation in that thread is included as part of the input sent to the LLM.

Fig 4. Short-term memory design

Given that this storage solution places the memory within the Cloud Run Service instance’s memory, any memory is lost when we release a new version of IBIS by re-deploying Cloud Run Service. For more details, you can refer to LangChain’s memory documentation.

Fig 5. Case for short term memory

Keep instance active

Since our short-term memory functionality currently stores memory data in the instance, we must keep this instance active to avoid memory loss during cold starts. To achieve this, we implemented a strategy based on the guidance from this document. We regularly send uptime checks to the Cloud Run Service instance to ensure it remains active. This approach is straightforward and incurs minimal cost. Additionally, we have restricted the scale-up of this service by setting both the maximum and minimum number of instances to one.

Conclusion & Future plan

Conclusion

The first release of IBIS was completed at the end of December 2024. Until the time I wrote this blog (Jan 2025), IBIS had been integrated into several key channels for handling incidents at Mercari. The number of users leveraging this tool continues to grow. We will consistently gather user feedback and monitor its impact on Mean Time to Recovery (MTTR).

Future plan

Accurately collecting user feedback is one of our core objectives. We plan to adopt a human-in-the-loop approach for automatic evaluations and gather user survey responses as data points to continuously enhance our product.
Transit from the traditional mention-based querying method to a Slack form-based questioning approach. This change is intended to improve the precision of responses by refining user queries.
Given the continuous updates to internal tools within the company, we plan to fine-tune our LLM model based on company documentation. This will ensure that the model provides the most current and relevant answers.

In Closing

Mercari, Inc. is actively seeking talented interns / new graduate Engineers, please feel free to explore our job description if you are interested.

How to bypass GitHub’s Branch Protection

Fri, 31 Jan 2025 14:14:30 GMT

Introduction

Hey everyone, my name is @iso and I’m working on the Platform Security Team at Mercari.

One of the major functions of our team is to ensure the security of Mercari’s GitHub code repositories with many different areas to consider in achieving this.

In this post, we’ll take a look at branch protection (protected branches) on GitHub; in particular, whether it’s possible for attackers to bypass rules requiring approval to merge pull requests. If you want to keep your branches safe, keep reading!

How we use GitHub at Mercari

Mercari uses GitHub to manage code. This includes not only app and backend code, but all sorts of files related to infrastructure, like files used for Terraform and Kubernetes. The data stored on GitHub plays a crucial role in our development process.

Different organizations may have different policies for GitHub permissions, but at Mercari, developers generally have write permissions for many repositories, including repositories used by other teams. (Of course, due to the nature of the content of some repositories, they are only accessible to a limited number of developers.) This means that developers can create new branches and pull requests (PRs) on other teams’ repositories or make pull requests that affect infrastructure in repositories that contain Terraform- or Kubernetes-related files.

While it’s convenient for developers to have write permissions for many different repositories, it’s not good if developers who have no affiliation with a certain repository can arbitrarily overwrite the code or modify important Terraform files without any form of review. That’s where branch protection rules and branch rulesets come in—with these rules, you can add a layer of security by requiring pull request reviews and approval before any changes can be merged into the default branch (main/master branch). At Mercari, we enforce branch protections for all repositories involved in production.

(Technically, branch protection rules and branch rulesets as used on GitHub have some differences, but for the purposes of this post, they’re functionally the same, so I’ll use the term "branch protection" to collectively refer to both.)

Methods attackers may use to get around branch protection

So now that we’ve established that branch protection plays a crucial role in protecting your repositories, what’s the best configuration to use? Can branch protection really protect your repositories from all types of attacks? Let’s find out!

Assumptions

Let’s assume the following simple conditions:

Situation: All developers that can access the repository have write permissions
Requirement: Changes to the main branch must be approved by at least one other person (= no developer can modify the main branch by themselves)
- In order to fulfill this requirement, let’s assume that the repository uses the branch protection rule "Required number of approvals before merging: 1"

Cast

To help us visualize each attack method, I’ll be walking through them using two characters.


Alice	A software engineer. Alice writes and reviews code on a daily basis. She has a keen sense of smell that can sniff out malicious code in code reviews, no matter how cleverly hidden it may be.


Mallory	An attacker. Mallory has big ambitions. She somehow acquired write permissions to a repository and is attempting to insert a backdoor in the code on the main branch.

The roles involved in a pull request

Before we get into the attack methods, let’s lay out how pull requests work and the different roles involved.

Pull requests are created by users or bots. I’ll refer to this person (or bot) as the "PR creator."

"Last commit pusher" refers to the user who pushed the most recent commit to the source branch (the merge base) of the pull request. In many cases, the PR creator is the last commit pusher ("PR creator" == "last commit pusher"), but this is not always the case.

Under the conditions we defined earlier in our assumptions, a pull request must be approved by at least one person. Let’s call this user the "PR approver." The person who created the pull request can’t approve it themselves, so we can say that in all cases, it holds true that the PR creator is not the PR approver ("PR creator" != "PR approver").

After a pull request is approved, it is merged into the main branch, but anyone with write permissions to the repository can merge the pull request. For the purposes of this post, it doesn’t matter who this person is.

Attack pattern 0: Mallory creates a pull request, and Alice reviews it

First, let’s think about the simplest attack method: Mallory creates a pull request that includes malicious code, and Alice reviews it.

As mentioned earlier, Alice’s keen sense of smell enables her to sniff out all malicious code in pull request reviews, so she finds the malicious code, rejects the pull request, and thwarts Mallory’s attack. This enables us to rule out all attack patterns in which Alice would be the PR approver.

PR Creator	Last Commit Pusher	PR Approver
Mallory	Mallory	~~Alice~~

Attack pattern 1: Mallory pushes a commit to a pull request Alice has created and approves the pull request (pull request hijacking)

This method is known as pull request hijacking. You can read more about it in this article:
https://www.legitsecurity.com/blog/bypassing-github-required-reviewers-to-submit-malicious-code

Pull requests can be approved by anyone (other than the PR creator) who has write permissions to the repository. This means that a malicious user could commit an arbitrary change to another person’s pull request, then approve and merge it themselves.

Alice may notice if a pull request she created has a commit added and is merged into the main branch, but if the pull request is created by a bot like Dependabot, it’s possible that no one will notice.

PR Creator	Last Commit Pusher	PR Approver
Alice	Mallory	Mallory

This attack method can be prevented by enabling the "Require approval of the most recent reviewable push" setting. Enabling this setting adds an additional rule requiring that the last commit pusher is not the PR approver ("last commit pusher" != "PR approver") meaning that Mallory won’t be able to approve the pull request.

Attack pattern 2: Mallory creates a pull request and uses GitHub Actions to approve it

In some repository configurations, a GITHUB_TOKEN automatically generated in a GitHub Actions workflow may be used to approve a pull request. Anyone with write permissions to the repository can create or add to a GitHub Actions workflow, so Mallory would be able to create a workflow to approve the pull request that she made.

When using a GITHUB_TOKEN to approve a pull request, the PR approver becomes "github-actions." This is treated as a separate user from Mallory.

PR Creator	Last Commit Pusher	PR Approver
Mallory	Mallory	github-actions

This attack method can be prevented by disabling the "Allow GitHub Actions to create and approve pull requests" setting. Disabling this setting adds an additional rule requiring that neither the pull request creator nor the pull request approver are github-actions ("PR creator" != github-actions && "PR approver" != github-actions).

Attack pattern 3: Mallory creates a pull request using GitHub Actions and approves it

In this attack pattern, similar to pattern 2, Mallory uses a GitHub Actions workflow to create a pull request and add code, and then approves the pull request herself.

PR Creator	Last Commit Pusher	PR Approver
github-actions	github-actions	Mallory

This attack method can be prevented the same way as attack pattern 2: by disabling the "Allow GitHub Actions to create and approve pull requests" setting.

Summary so far

Let’s summarize the attack patterns we’ve described so far, as well as other possible patterns.

In the table below, countermeasure 1 and countermeasure 2 are defined as follows:

Countermeasure 1: Enable the "Require approval of the most recent reviewable push" setting
Countermeasure 2: Disable the "Allow GitHub Actions to create and approve pull requests" setting

Attack Pattern	PR Creator	Last Commit Pusher	PR Approver	Can this be prevented with countermeasure 1?	Can this be prevented with countermeasure 2?
1	Alice	Mallory	Mallory	✅ Yes	❌ No
2	Mallory	Mallory	github-actions	❌ No	✅ Yes
3	github-actions	github-actions	Mallory	❌ No	✅ Yes
4	github-actions	Mallory	Mallory	✅ Yes	✅ Yes
5	Mallory	github-actions	github-actions	✅ Yes	✅ Yes
6	Alice	Mallory	github-actions	❌ No	✅ Yes
7	Alice	github-actions	Mallory	❌ No	❌ No

Attack pattern 7: Mallory adds a commit to a pull request Alice has created using GitHub Actions and approves it herself

Attack patterns 1–6 can be prevented by changing the settings on GitHub. However, unless we change the assumed conditions, there doesn’t appear to be a way to prevent attack pattern 7.

In this pattern, Mallory uses GitHub Actions to add malicious code to a pull request created by Alice. Mallory then approves and merges the pull request herself. (The pull request that Mallory adds code to using GitHub Actions doesn’t need to be a pull request created by Alice. It could be a pull request created by a bot like Dependabot or an open pull request that has been long forgotten. In either of these cases, it’s unlikely anyone would notice the attack.)

PR Creator	Last Commit Pusher	PR Approver
Alice	github-actions	Mallory

How to prevent attack pattern 7

In this attack pattern, the PR creator, last commit pusher, and PR approver are all different users, enabling Mallory to bypass the settings we’ve discussed so far.

The method GitHub offers to prevent this attack is to set the required number of approvals before merging to 2 or more. However, increasing this number lowers developer productivity and is not a great solution.

Enabling the "Require review from Code Owners" setting can make it harder for an attacker to use this attack pattern, but if Mallory is a code owner, she can always bypass the setting. This setting may lower the success rate of attacks, but it can’t prevent them entirely.

Currently, it isn’t possible to prevent this attack using just features provided by GitHub, so in order to close off this attack pattern, it’s necessary to develop some sort of mechanism yourself. Some possible examples:

Create a mechanism that raises an alert when a pull request that looks like the one in attack pattern 7 is merged
Set the required number of approvals before merging to 2 and have a bot approve the pull request if it doesn’t look like the one in attack pattern 7; this will enable a pull request to be merged with approval from one person and a bot

Here, I should note that I notified GitHub about the lack of features that would prevent this attack pattern in May 2024. GitHub responded saying that this is expected behavior. They also gave permission for me to publish this blog post.

Conclusion

In this post, we covered branch protection on GitHub, methods an attacker might use to evade branch protection, and countermeasures that can be taken to prevent those attack methods. Branch protection is a powerful feature that can be used to protect important branches, but it isn’t perfect; under the right conditions, it can be bypassed using GitHub Actions. I hope this information helps readers use GitHub more securely in both their personal and work repositories.

JSNation and React Summit 2024 US Participation Report

Thu, 26 Dec 2024 12:34:00 GMT

Hello, I’m @tanasho, a Software Engineer at Mercari. I typically work on developing Mercari Hallo. At Mercari, we have a system in place that supports individual growth, as described in the follwing article. Recently, I took advantage of this system to attend the JSNation & React Summit 2024 in the US in person.

メルカリのエンジニアリングカルチャーについて

In this article, I would like to share not only the technical aspects of the conference but also the atmosphere at the venue and my unique experiences, such as being able to discuss with the speakers in person. I hope this report will be helpful for those considering attending a frontend conference in the future.

What are JSNation and React Summit US 2024?

JSNation and React Summit are conferences organized by GitNation. JSNation focuses on JavaScript, while React Summit focuses on React. These events also cover related technologies like Next.js and AI, as well as soft skills like collaboration among engineers. They also provide many opportunities designed to foster networking among engineers, such as lunchtime meetups, workshops led by speakers (held on a separate day), and interactions at company booths. A combo ticket that allows you to attend both events, so I used that to participate.

Here is the schedule and location for each in-person conference.

Date Time (EST)	Event	Location
2024/11/18 9AM – 5PM	JSNation US 2024	Liberty Science Center
2024/11/19 9AM – 5PM	React Summit US 2024	Liberty Science Center

I’ll go into more detail about each of the events below.

JSNation 2024 US

The presentation took place at two venues, with the main stage located inside a planetarium!
I would like to highlight a session that caught my attention there.

Session – JavaScript Evolution and Updates

JavaScript Evolution and Updates

This presentation covered the latest JavaScript methods and the "Baseline" project.
Baseline provides information on browser support for web features such as Javascript methods.

In our frontend development, it can be difficult to ensure there are no issues when using new JavaScript methods in production by checking resources like MDN or “Can I use.” Our team has also been discussing and exploring ways to automatically detect if new Javascript methods can be safely used in all core browsers during the coding phase, using tools like a linter. That’s why I was interested in this presentation.

Baseline helps with compatibility checks in all core browsers at the following stages.
As described in the website linked below, core browsers include not only desktop browsers but also mobile browsers.

Newly available: The feature is now supported by all core browsers.
Widely available: 30 months (2.5 years) have passed since the feature became compatible across core browsers.

https://web.dev/baseline

Deciding which state to use depends on your project. Given that defined “all core browsers” are widely used today for both desktop and mobile, Baseline serves as a reliable indicator that can help in compatibility checks.

Moreover, there is consideration for Baseline supporting developer tools like a linter. While we’re not sure what these tools will look like yet, I’m excited to imagine a future where a linter can automatically meet Baseline requirements during the coding phase.

After the session, there was Q&A time where participants could ask a wide range of questions, from casual topics to technical queries.

Question Room

There was also a Question Room where attendees had the opportunity to talk with the speaker immediately after the session. I visited the Question Room after the JavaScript Evolution and Updates session to talk with the speaker. It was a great opportunity to deepen my understanding of the session, and I was delighted to connect with the speaker.

During our conversation, we discussed a real issue related to the portal application we typically develop for our partners. In this application, we use the crypto.randomUUID() method and encountered an issue when a portal user accessed it with an older version of the Safari browser on a PC. We talked about how beneficial it would be to have a system that allows developers to specify target browser versions and to detect any code that doesn’t meet these version requirements at the coding phase using a linter.

I also enjoyed hearing about life in New York and how the presenter, a member of the Chrome team, works. This made the day a very meaningful experience.

React Summit 2024 US

The React Summit also featured presentations held in two venues, similar to JS Nation. At that time the release of React 19 was approaching, so there were presentations introducing its new features and panel discussions on the future of React. I would also like to share the atmosphere at the React Summit.

Session – Aligning Patterns Across Design and Development

Aligning Patterns Across Design and Development

This was an introduction to Code Connect, the latest feature in Figma. I participated because I wanted to explore if I could leverage Figma more efficiently in my work.

Code Connect is a feature in Figma designed to bridge the gap between designers and engineers. With this feature, you can reflect the implementation of components in Figma’s design, enabling synchronization between code and design in Figma.

Code Connect

Moreover, prop names are also integrated, providing a unified understanding of which props are used to display the component. This setting code can be viewed and copied directly from the Code Connect panel in Figma.

Taken from https://www.figma.com/code-connect-docs/quickstart-guide/

In my work, I sometimes find it confusing to choose the correct props in the design system because it’s not immediately clear which props are being applied. Additionally, there are cases where the prop names differ between implementation and design. While many methods exist to generate code from design, they don’t always sync perfectly. This feature is interesting because its approach of reflecting code in design ensures consistent synchronization and aligns understanding between designers and engineers.

Additionally, linking component code to Figma is straightforward thanks to the interactive setup command as described in the guide below. To summarize, this command generates a figma.tsx file for the component and then you use a publish command to sync it with Figma.

Getting started with Code Connect

Sponsor Booths

Several sponsor booths were set up at the venue.
After the presentation ended, I visited Figma’s booth and engaged in various casual conversations with the people from Figma through demos. During the demos, it occurred to me that the feature was similar to Storybook, so we discussed how to effectively differentiate usage between Storybook and Code Connect. We talked about the fact that Code Connect is for aligning understanding of UI components between designers and engineers, while Storybook seems to be for checking their actual behavior and testing components.

In conclusion

This concludes my report on JSNation and React Summit US 2024. Although this article doesn’t cover every presentation, a variety of interesting topics were discussed, such as the use of Memlab for detecting memory leaks and the introduction of an AI-powered Chrome Inspect tool. One of the great aspects of attending conferences like this in person is the opportunity to explore these topics further during Q&A sessions, connect with fellow engineers, experience demos, and share our daily technical concerns and interesting technologies through casual discussions.

If you’re interested and planning to attend, I suggest preparing a schedule ahead of time to select which sessions to join as there are many choices and time is limited. JSNation and React Summit are also held in the Netherlands, so you could consider attending there as well.

How to unit-test Mercari Hallo Flutter app

Tue, 24 Dec 2024 11:00:12 GMT

Introduction: Embracing Unit Testing in Flutter

Hi, I’m Heejoon, a software engineer at Mercari. I’m part of the Work Mobile team working on the Mercari Hallo app. I’m excited to share our approach to unit testing—it’s a big part of how we build a high-quality app!

Unit testing is essential for modern software development, especially for Flutter apps. It’s all about testing individual parts of our code (functions, classes, widgets—anything and everything!) in isolation to make sure they’re working as expected. Think of it like checking each ingredient of a recipe before baking—it helps avoid a disaster! By verifying that each piece works correctly on its own, we build a rock-solid foundation for a reliable and maintainable app.

So, why is unit testing so important in the ever-evolving world of Flutter?
Let’s look at some of the benefits we’ve found:

Early Bug Catching: Unit tests are like our bug-catching superheroes. They find problems early in the development process, saving us headaches down the road.
Better Code Design: Writing unit tests helps us design our code better. It encourages us to think about how different parts of our code work together, leading to more organized, understandable, and reusable code.
Refactoring Without Fear: Refactoring is like cleaning up our code—making it more efficient and easier to work with. Unit tests give us the confidence to refactor without worrying about breaking things. They’re our safety net!
Faster Development (Really!): We know writing tests might seem like extra work at first. But trust us, it actually speeds up development in the long run. By finding bugs early and making refactoring easier, we build features faster and with more confidence.

While other types of testing (like integration tests) are important, we’re focusing on unit and UI testing in this article. We’ll walk through how we write effective tests for both our UI and business logic, sharing practical tips to help everyone build robust Flutter apps.

Setting Up Your Flutter Testing Playground

Getting started with testing in Flutter is super easy, thanks to the awesome flutter_test package that’s already built-in! Here’s how we set up our testing lab:

Add the secret ingredient: In your pubspec.yaml file, add flutter_test as a dev dependency. It’s like adding superpowers to your project!
Power up your project: Run dart pub get. This grabs the flutter_test package and all its helpful sidekicks.
Build your testing arena: Create a new file (something like widget_test.dart or logic_test.dart) inside a test directory at the root of your project. This is where the testing magic happens! ✨

Unit Testing

How to Test Simple Logic

Thoroughly testing core application logic, separate from the UI, is crucial for building robust and maintainable Flutter apps. This involves testing pure Dart code, such as models, services, and utility functions. Let’s illustrate with a practical example from our production codebase:

This code defines a Fraction type extension that converts a fractional value to a percentage, rounding up. The doc comments now include illustrative examples.

Now, let’s write unit tests to verify its behavior:

To understand how these tests function, let’s break them down:

The group('asPercentage', () { ... }); block organizes related tests, improving the clarity of our test output. Think of it as categorizing our tests.
Each test() function defines a specific scenario. The first argument is a descriptive label, and the second is the test logic.
expect(actualValue, expectedValue); asserts that our asPercentage method’s output matches the expected value. Any mismatch signals a potential issue.
Our test suite covers various scenarios, including different decimal places, boundary values like zero and one, and negative inputs. This comprehensive approach ensures the reliability of our asPercentage method.
Note how our tests include boundary values (zero and one) and negative input. Testing these edge cases is crucial for uncovering hidden bugs and ensuring our function behaves correctly in all situations.

These tests also demonstrate key principles of effective unit testing:

Descriptive Test Names: Clear test names act as documentation, aiding our understanding and maintenance. For example, we are encouraged to choose "rounds up to the nearest integer with no decimal places" over "test case 1".
Structured Test Organization: Using group() categorizes our tests for improved readability and navigation.
Comprehensive Coverage: Testing various inputs and edge cases strengthens the robustness of our code.
Adhering to Conventions: Our test file name (fraction_test.dart) follows the convention of appending _test and we put it into the same file path as the production file path just replacing "/lib" with "/test", which aids in organizing our tests.

By following these practices, we create effective unit tests that enhance the quality, reliability, and maintainability of our application.

How to Test Time-dependent Logic

Here’s another example that tackles a common challenge: dealing with time in our tests. We’ll focus on how we display elapsed time in a user-friendly way.
Imagine you want to show users how long ago something happened, like "5 minutes ago" or "2 days ago."

We use a Riverpod provider called elapsedTimeFormatProvider for this:

This provider takes a DateTime (target) and returns a human-readable string (e.g., "5 minutes ago"). We leverage Riverpod for dependency injection.

Now, here’s the key for testing: clock.now(). Typically, you’d use DateTime.now() to get the current time. But in tests, DateTime.now() presents a problem: it’s always changing! This makes our tests unpredictable. We want our tests to produce the same results every single time, no matter when they run. This is what we call deterministic tests.

The clock package solves this problem. It lets us freeze time and set it to a specific point. This gives us complete control over time in our tests, which is essential for writing reliable and consistent unit tests.

This test case shows a neat trick for dealing with time in our tests—something that can be a real headache! That’s where the clock package comes in, with its trusty sidekick withClock. Check it out:

We’re using Clock.fixed(baseTime) to create a magical frozen clock. We set baseTime to a specific moment (April 17, 2024, at 10:00:00 in this case). Time stands still inside that withClock block. Any code that calls clock.now() will get our baseTime, not the actual current time.

So, what’s the big deal? Well, it means our tests become deterministic. They’ll give us the same results every time, no matter when we run them. No more flaky tests due to the ever-ticking clock!

Inside the withClock block, we call our time-formatting provider (elapsedTimeFormatProvider) with different dates and check that it gives us the right strings (like "1 second ago," "59 minutes ago," and so on). Since time is frozen, we know exactly what to expect.

This trick is a lifesaver for testing time-based logic. The clock package and withClock, along with Clock.fixed, give us the power to control time in our tests, making them super reliable. It’s a must-have in your Flutter testing toolkit!

We’ve all been there: spending hours debugging a flaky test only to realize it’s because of DateTime.now(). To prevent that pain, we use a custom linter that guides us toward clock.now() instead. It’s a simple way to avoid those time-related testing headaches. We’d love to talk more about our custom linters—they’re pretty cool—but that’s an adventure for another day!

Widget Testing

Alright, so we’ve tackled the nitty-gritty of testing our backend logic. Now, let’s move on to the exciting part: ensuring our Flutter UI looks and behaves exactly as we envisioned! Widget testing, sometimes referred to as component testing, lets us verify the appearance and functionality of individual widgets, guaranteeing they render correctly with various inputs and states. This proactive approach helps us squash those pesky UI bugs before they reach our
users and potentially lead to negative app store reviews.

So, how do we put our widgets to the test? Flutter provides a handy testWidgets() function specifically for this purpose. It creates a simulated environment where we can render our widget, interact with it (e.g., tapping buttons, entering text), and then verify its behavior.

Here’s a simple example of a typical widget test:

However, our widget tests often look a bit different in practice. We’ve implemented some custom wrappers to streamline our testing process and handle the complexities of our app’s architecture, which uses Riverpod for state management. A more representative example of our tests would be:

Here’s a breakdown of our custom functions:

testThemedWidgets(): This wraps testWidgets() and runs the test multiple times with different combinations of light/dark themes and surface sizes (defined in surfaceSizes). It also tags these tests with 'golden' to facilitate efficient golden image updates using the command flutter test --update-goldens --tags golden.
pumpAppWidgetWithCrewAppDeps(): This wraps pumpWidget() and handles the setup of necessary Riverpod providers, simplifying the boilerplate required for each test.
matchesThemedGoldenFile(): This wraps matchesGoldenFile() and, in addition to performing the standard golden file comparison, it dynamically replaces placeholders like {theme} and {size} in the filename with the actual values used during the test run.

By running flutter test --update-goldens --tags golden, we generate four
golden images: golden/light-320x480/my_widget_test.png, golden/light-375x667/my_widget_test.png, golden/dark-320x480/my_widget_test.png, and
golden/dark-375x667/my_widget_test.png. These images, along with the test
code, are committed to version control to prevent unexpected visual regressions.

Code Coverage

We love writing tests! But how can we be sure we’ve written enough? Code coverage helps answer that question. It tells us the percentage of our code executed during tests, allowing us to identify gaps in our testing strategy, ensure critical code isn’t left untested, and even uncover dead code. Think of it like exploring a treasure map—you don’t want to leave any areas uncharted!

We’re especially interested in coverage changes with each pull request. This verifies that the new code is well-tested and that existing tests remain effective.

Our CI/CD pipeline completely automates code coverage analysis:

Generate Report: The pipeline runs flutter test --coverage, producing a detailed report (coverage/lcov.info) showing executed code lines.
Clean Report: The pipeline refines lcov.info, removing irrelevant entries (like generated code) for greater accuracy using commands like:
Generate Visual Report with Coverage Metrics: The pipeline uses genhtml to create a user-friendly HTML report from the (filtered) lcov.info:

This generates an HTML report displaying both overall and differential coverage (changes introduced by new code). Differential coverage, inspired by the paper "Differential coverage: automating coverage analysis", helps pinpoint areas needing more tests and ensures existing coverage isn’t negatively impacted.
Upload Report to Cloud Storage: For easy access, the pipeline uploads the HTML report (with differential coverage) to a Google Cloud Storage bucket, enabling convenient browsing.
Summarize Coverage in Pull Request: The pipeline adds a concise coverage summary to the pull request, including a link to the HTML report in Cloud Storage. This lets reviewers quickly assess coverage changes.

This automation streamlines our workflow and maintains high test quality, giving us confidence in our codebase and allowing us to focus on building great software.

The screenshot above shows a real coverage summary. We’re continually working to improve these reports! What do you think?

Advanced Topics

While we strive for comprehensive testing, sometimes we encounter roadblocks. Let’s briefly touch on several common challenges:

Defining the "Unit": In a Flutter context, deciding what constitutes a "unit" for testing can be nuanced. We aim to test individual widgets and their associated business logic in isolation, but the level of granularity can vary. Sometimes, testing a small group of interconnected widgets as a unit makes more sense than strictly isolating every single widget. Finding the right balance is key to effective unit testing.
Legacy Code: Even in a relatively young codebase like ours, some early-stage code can be difficult to test. This often stems from initial rapid development prioritizing features over testability, resulting in tightly coupled components and complex dependencies that make writing tests challenging. Refactoring these areas can improve testability, but requires careful planning.
Mocking Dependencies: Testing components that rely on generated custom hooks from graphql_codegen, particularly those interacting with the GraphQLClient from the graphql package, presents a unique mocking challenge. Effectively isolating our logic for testing requires carefully mocking both the client and the generated hooks, which can become complex depending on the query structure and data flow. Tools and techniques for mocking these specific dependencies are crucial for robust testing.

This section is intentionally brief; a deeper dive into these topics warrants dedicated articles in the future. Stay tuned!

Wrapping Up: Unit Testing for a Robust Mercari Hallo

That’s a wrap on our unit testing journey! We’ve covered a lot of ground, from setting up your testing environment to tackling tricky scenarios like time-dependent logic and mocking dependencies. We’ve also shown how we leverage custom tooling and CI/CD integration to streamline our testing process and maintain high code coverage.

Hopefully, this deep dive into our unit testing practices at Mercari, specifically for the Mercari Hallo app, has provided you with valuable insights and practical tips you can apply to your own Flutter projects. Remember, unit testing isn’t just about finding bugs; it’s about building a solid foundation for a robust, maintainable, and scalable app. It’s an investment that pays off in the long run with increased developer confidence, faster development cycles, and ultimately, a happier user experience for Mercari Hallo users.

We hope this article has been helpful to your projects and technical explorations. We will continue to share our technical insights and experiences through this series, so stay tuned. Also, be sure to check out the other articles in the Mercari Advent Calendar 2024. We look forward to seeing you in the next article!

Leading a project to migrate hundreds of screens to SwiftUI/Jetpack Compose from UIKit / AndroidView in Merpay

Tue, 24 Dec 2024 10:00:15 GMT

This post is Merpay & Mercoin Advent Calendar 2024 , brought to you by the Merpay Engineering Manager @masamichi.
The Merpay mobile team is currently working on a project to migrate hundreds of Merpay screens that exist within the Mercari app to SwiftUI/Jetpack Compose.
This article describes the history of the project and how it is proceeding.

Release of Merpay

The Mercari app with Merpay was released in February 2019. The initial development was mainly done in 2018.At that time, SwiftUI and Jetpack Compose were not announced, and the Mercari app with Merpay was developed in UIKit/Android View.
SwiftUI/Jetpack Compose, a declarative UI framework for iOS and Android, was then announced within 2019.

GroundUP App Project

Meanwhile, around 2020, the mother app, Mercari app, launched the GroundUP App project to revamp its code base to solve issues that had accumulated over years of development.
The GroundUP App project fully adopted SwiftUI/Jetpack Compose and was ready for release in 2022.

For more details on the project, please refer to the core members articles:

Various functions of Merpay were modularized and embedded in the Mercari app in a somewhat loosely coupled state, so we were able to make them embedded in the new app and continued to develop new features in parallel with the GroundUP App project.

For more information on the Merpay migration, please refer to these articles:

DesignSystem

Mercari has defined DesignSystem for screen design and development. Mercari has been gradually introducing it to the app since around 2019.
In particular, the new app after the GroundUP project has been revamped with SwiftUI/Jetpack Compose-based UI components, and the full adoption of the DesignSystem has resulted in a unified screen UI/UX, dark mode support, and improved accessibility.

On the other hand, as mentioned above, Merpay has integrated the modules developed since the beginning directly into the new application. The screens were based on UIKit/Android View, and the DesignSystem was also based on the previous version of the UIKit/Android View-based implementation. As a result, there were issues such as differences in UI/UX, lack of dark mode support, and architectural differences due to the different UI frameworks.
In order to take full advantage of the benefits gained from the GroundUP project, a project to migrate Merpay existing screens was started in 2023.

Engineering Projects and Golden Path

Migrating hundreds of Merpay screens requires a long-term commitment. Merpay has developed a framework called Engineering Projects to drive these long-term engineering investments.
For more information on Engineering Projects, please read this article by @keigow, VP of Engineering.

メルペイのエンジニアリングへの投資を推進する仕組み

We have also defined a standard technology stack across the entire Mercari Group as the Golden Path, aiming to improve development efficiency and reuse of technology assets. Merpay migration project is called the DesignSystem Migration Project for simplicity.

Building an Engineering Organization That Promotes Global Expansion—Meet Mercari’s Leaders: Shunya Kimura / CTO

The actual implementation of migration requires man-hours and discussion of priorities. I have prepared a project plan to launch this project, clarifying the background, actions, structure, and milestones. We are promoting this project as one of the Engineering Projects.

Project Structure and Approach

Structure

Merpay has a cross-functional team structure that includes product managers and engineers for each of the major program domains. Proceeding with the Design System migration involved collaboration between mobile teams and designers from all of the programs. Regular bi-weekly meetings with mobile team leaders and designers were held to share progress, blockers, and regularly set milestones. During the project launch phase, a weekly meeting cadence was adopted. Once the project was somewhat solidified, the frequency of meetings became bi-weekly.

I created an internal Confluence page with all of the project information. This Confluence page included the project plan, structure chart, Slack communication channels for each function, design and development know-how, QA test cases, feature release status, regular meeting minutes, and other information necessary for the project to be viewed from a high level.

Excerpts from the Table of Contents

Man-hours and timing are important to proceed with migration. Migration can be carried out efficiently if it can be done at the same time as the introduction of new measures for the product. On the other hand, this alone will not allow migration of functions that change little. In addition, there are cases where development with a high degree of urgency is temporarily carried out on existing screens in order to prioritize speed. We work closely with the design and mobile team leader of each program to ensure a good balance between migrating existing functions as they are and migrating at the same time as new product initiatives are introduced.

Screen List and Progress Tracking

In order to migrate screens, it is first necessary to understand as accurately as possible how many functions and screens there are. At Merpay, we created a spreadsheet with a list of all the screens when we started the project.This allowed us to accurately identify the number of screens and screen patterns, as well as the team, development, and design staff with ownership of the functionality in a centralized location. We have also assigned IDs to all screens to ensure that there are no discrepancies in the recognition of the target screens within the team.

Each screen is also assigned a progress status as shown below and plotted on a graph so that the overall progress can be visually tracked.

TODO
Design In Progress
Design In Review
Design Done
Dev in Progress
In QA
Done

We update the progress of the features we are working on for migration at our regular bi-weekly meetings.
By accurately tracking the status of each screen, we are able to report transparent and accurate information to the CTO and VPoE at regular Engineering Projects meetings.

Excerpts from the Screen List sheet

Strategy Sharing

At Merpay, we have a quarterly event called Strategy Sharing, where we review and share the company’s strategy and roadmap, as well as deciding on the priorities for the next quarter. During this event, we also share Engineering Projects milestones with the whole company. This allows people outside of the engineering department to understand how the project is progressing and gain recognition throughout the company.

Once in the second half of each quarter, Merpay holds a “Strategy Sharing,” in which priorities for the next quarter’s initiatives are decided and the strategy and roadmap are reviewed and shared with the entire company. During this process, we define the functions and progress rates to be targeted in the next quarter and share milestones company-wide about Engineering Projects. This allows people outside of the engineering department to track progress and gain company-wide recognition.

Current Progress

We have been promoting the project for about two years from 2023 to 2024, and as of December 2024, Android has completed about 65% of the migration and iOS about 60% of the migration, and it has been released. Including those under development, migration is progressing at 70% ~ 80%.

Android Progress

iOS Progress

Our team will continue to work together to update Merpay’s mobile engineering, and we will continue to promote the project with the goal of 100% completion.

In Closing

This article introduced the background and approach we took to migrate hundreds of Merpay screens within the Mercari app to SwiftUI/Jetpack Compose. The project has been a large, long-term effort filled with difficulties, but I believe that tackling this kind of challenge is a testament to the Mercari Group’s strength as an engineering organization. I hope this article will be helpful to all teams considering or in the process of migrating to SwiftUI/Jetpack Compose.

Next article will be by @kimuras. Look forward to it!

Good tools are rare. We should make more!

Mon, 23 Dec 2024 18:01:42 GMT

Most tech companies are full of different custom helper tools. I don’t even mean “big” tools — like frameworks, libraries or programming languages. Think about the little apps we all use to help with debugging or creating test objects. Or your Feature Flag management system — or the inspection tools that Customer Support uses to help your users.

It’s rare that these tools are exciting, and it’s not often they are appreciated or much cared for either.

This is understandable — in some ways, I don’t want my tools to be exciting. I want them to let me do what I need to do, and allow me to get on with my day. From a certain angle, I want them to be invisible.

We need good tools. We deserve good tools! Our users deserve us having good tools.

So what makes a tool good?

Here are a couple of guiding principles that I think are helpful to keep in mind when working on your tools:

Accessible

I think this is simultaneously the easiest and hardest thing to get right. Usually, when working on tooling, we’re hyper-focused on a specific problem.

This makes it easy to also make a hyper-specialized tool that requires a lot of project/team/domain specific knowledge to be able to use well.
To some degree, this is inevitable — if you’re working on a tool that helps with managing microservices, people using the tool need to have a concept of what a microservice is!

But there’s an opportunity here!

Can you make that accessible to people whose day-to-day life doesn’t revolve around Kubernetes, Helm, and Terraform? Can you make your tool hide some of the underlying complexity?

Can you simplify adding a new service, so that a mobile engineer can spin up an experiment easily? It’s not easy, but it’s work that often pays off in the long run.

Easy to use

Another aspect of this is also just making things pleasant to use.

If your underlying model for a field technically accepts arbitrary strings, but 95% of values are gonna be literally “true” or “false” — provide affordances for that. A simple toggle or a button that preselects one of the values is very simple to add, but makes the interactions so much more pleasant.

Typing in “true” once isn’t the end of the world. However, making tens or hundreds of your coworkers do it multiple times a day isn’t great.

Another often forgotten aspect — performance is also an important feature.

You probably don’t have to sweat every last millisecond, but if your tool takes 10s to load a simple list, it’ll be frustrating to use.

Working on making things faster is one of the easiest ways to get into the good graces of your fellow engineers. This extends doubly so to anything used directly when interacting with code — there’s no easier way to slow the company down, than by adding a couple seconds on a critical path when rebuilding the app. Shaving those seconds from an existing state will make you a hero.

Complete

A corollary to the previous principle is that one of the easiest things to make your tools easier and nicer to use, is to just make them do more.

Maybe it’s just my personal pet peeve, but nothing takes me out of a flow quicker than having to jump from tool to tool.

You want to let people stay in one place as much as possible. If you’re working on, let’s say for a sake of argument, a sort of a Marketplace, where people sell and buy items? Let them create new test accounts, fund them, create new items, create transactions, change shipping statuses, send reviews, etc — all from one place. Can you imagine how tiresome it would be if doing each of these steps required you to use a separate tool?

You have to, of course, put the limit somewhere — you don’t want to end up with an unmaintainable kitchen sink of utilities that is impossible to navigate and maintain.

In my opinion, however, that line is probably higher than most people think.

Open

In a company full of engineers, you’ll very quickly have people being annoyed by perceived deficiencies in your tools.

Some of those will just complain to colleagues — but some of them will eventually get fed up with the problem. They’ll try to take things into their own hands and improve the tools, even though they’re not owned by their team.

This is the best thing that could happen to you. Your tools are now better, and you didn’t have to lift a finger.

Alas, engineers are territorial and opinionated creatures.

This is a controversial stance, but I think unless something is an egregious pile of hacks — if it makes the experience of using the tools unambiguously better, you should just accept the changes.

It doesn’t matter if the ~ vibes ~ of the codes are off, if you’d have architected it slightly differently, if you don’t like how the strings are named.

Is the tool better with the PR than without, and likely won’t cause immediate problems? Accept it.

It’s of course absolutely fine to have feedback, and suggest improvements! But if they’re not absolute deal-breakers, they shouldn’t block landing the change.

What it boils down to is: The barrier to accept changes to your own tools should be lower than to the code you’re shipping to customers. If it’s the other way around, something is wrong.

There are, of course, times when this is unfeasible, or tools need to be closely controlled and guarded for security and/or audit reasons — but thankfully, for the vast majority of situations, that’s not the case.

Make your tools easy to contribute to, write basic docs, and your tools will soon start improving without your involvement.

Extendable

This is corollary to the “completeness” argument — your team will never predict all the use cases or issues other teams will hit. It’s great if your architecture allows people to layer their own customization on top of your own tools. But it’s also fine for simple things to be duplicated and live in multiple places.

Think about feature flags — there’s always some “canonical” place to add overrides and whitelist yourself for tests or development. But that very well could (and should!) live inside the app too!

Making a simple interface to allow people to add local overrides takes a couple of hours, but it will very, very quickly pay for itself by people being able to just stay in the app when testing something, without having to jump back and forth between the browser and the app.

Your internal tools aren’t programming languages — it’s fine for there to be more than one way to do something.

Not forever

This is probably obvious to some, and sacrilegious to others.

Tools you build don’t have to be temporary, but it’s fine if they are. If they have served their purpose, it’s fine to let them go.

On the flipside, it’s also completely fine to build them knowing that they will be obsolete soon!

Let’s imagine you’re waiting on a sibling team to finish their API. The API not being deployed makes testing your UI much harder, because you don’t have an easy way to get your app into the required state.

If the surface area of your UI is big, it might make sense to add a little helper inside the app to completely ignore the API, and just set up the correct properties manually.

It might be obsolete in two weeks when the API actually shipped, but you have made more progress in the meantime by not being blocked or slowed down by the lack of it.

Most things in life are temporary. It’s fine for code to be too.

Cost of bad tools

So what’s the worst that can happen when your tools get neglected, or are never cared for in the first place?

Every chef and woodworker knows that blunt tools are dangerous. A blunt knife is more dangerous than a sharp one because it’s unpredictable. You know exactly what a razor-sharp knife will do, and you can position yourself to mitigate any danger.

Thankfully, working on software rarely has catastrophic failure modes of losing a finger; but bad tools can still be costly – but not always in obvious ways.

When a tool is unreliable, slow, or just straight up buggy — it’s very easy to notice (and measure!). But sometimes some things are just unpleasant, or tedious to do. It’s easy to dismiss those — “oh, it’s just an unfinished UX”. But those can be damaging in the long run too.

Having to jump between five different apps — some of them in Slack, some documented in Jira, some living in an internal portal, some requiring extra third-party apps open is not free. Every new interaction adds that little extra bit of cognitive load, that little extra bit of friction. None of them feel like a big deal in isolation, but they do add up pretty quickly!

People have different tolerances for tedium; but everyone has a breaking point. The idea of testing another potential edge-case is unbearable, because it requires clicking through 10 different dialogs, and things get overlooked. It’s death by a thousand paper-cuts.

So how do good tools look in practice?

My favorite improvement this year was adding a completely new debugging layer in our iOS and Android app. We’ve had an internal debug menu; but recently when working on Hassle-Free Car Sales we’ve extended it to be helpful on that project specifically.

This project was, in fact, one of those mentioned above — the client engineers had a couple of weeks of head-start, compared to backend. We very quickly decided to focus on getting the UI right, and leave integration with actual backend services to the very end. We had a rough idea of what the API shape will be when starting, but didn’t spend time focusing on the details until much later.

To let us effectively work on it, my teammates added a sub-menu that let us ignore the network entirely, and just override required properties directly in the app.

This shaved weeks from the project time — we were able to test and QA a good chunk of client-side code, before a single backend service was ready.

The override menu being directly in the app also encouraged us to test things more thoroughly — being able to toggle between all the different states without ever leaving the app dramatically reduced how much friction it took.

Other things we made better this year include significantly cutting down on the amount of disk space and time that our iOS unit test takes; making the UI for the Feature Flags much nicer to use, and adding an on-device visualization of all the analytics calls we make.

That work wasn’t always easy or pleasant — but it has universally paid off.

That’s of course only a small (and very mobile-centric!) chunk of the work we’ve done — and there’s more on the way to ship in 2025.

Hope you’ve had a good 2024, and wishing you the best (tooling) in 2025!

A smooth CDN provider migration and future initiatives

Mon, 23 Dec 2024 11:00:46 GMT

Introduction

Hello! I’m hatappi from the Microservices Platform Network team.

Since 2023, Mercari has been gradually migrating our content delivery network (CDN) provider from Fastly to Cloudflare. We have completed the traffic migration for almost all existing services, and all new services are now using Cloudflare.

In this article, I will focus on the migration process itself, not on comparing CDN providers, while explaining the approach we took to ensure a smooth migration. I will also introduce our internal "CDN as a Service" model, which is the ultimate goal of our CDN efforts.

Background

At Mercari, our network team has managed hundreds of Fastly services across both development and production environments. Our team also maintains Cloud Networking like a GCP Virtual Private Cloud (VPC) and Data Center Networking. We needed to find a way to conduct the migration smoothly within given time constraints.

Migration Steps

Preparation

Though both Fastly and Cloudflare are CDN providers, they do not behave in exactly the same way. For example: Fastly separates cache respecting the origin’s Vary header, but Cloudflare currently only supports this for images. We needed to investigate which features were being used in Fastly and how to implement them in Cloudflare.

We focused on not significantly altering the current behavior when considering migration features. Starting a migration might lead to adding improvements or trying new features. Such an approach could be manageable for a few services, but attempting to apply it to hundreds of services would make the migration endless. Therefore, keeping the migration scope narrow was crucial for a smooth migration. This philosophy helped in subsequent steps as well.

Implementation

We use the official Terraform provider to manage Cloudflare. Instead of using Terraform resources individually for each service, we created a Terraform module with the necessary functionality within the module required to reuse it in upcoming service migrations.

In Fastly, the logic we implemented and Fastly’s logic gets compiled into a single VCL (Varnish Configuration Language) file. Initially, we manually checked each VCL and implemented changes into Cloudflare’s Terraform resources, which took more than 30 minutes per implementation.

However, as more services were migrated, we found certain classes in the VCL logic; necessary migration logic, and ignorable logic. Therefore, in the later stage, we developed migration scripts using Go, automating the Terraform module settings based on VCLs. Any logic that couldn’t be automatically configured was shown as output. This allowed us to complete implementations for simple services in just a few minutes.

Testing

Most services have both development and production environments, so we tested in the development environment before migrating production. For services with high traffic or mission-critical features, we wrote code to test behavior beforehand. Since we didn’t drastically change behavior from Fastly, we could write tests comparing against Fastly service behavior, allowing confident commencement of traffic migration.

Traffic Migration

Regardless of the number of tests conducted, actual traffic migration requires caution, especially ensuring smooth rollback in case of issues.

We adopted an approach to meet these requirements at the domain name system (DNS) layer. Mercari uses Amazon Route 53 and Google Cloud DNS, both of which support weighted routing. This allows us to gradually migrate traffic from Fastly to Cloudflare. In case of issues, setting Cloudflare’s weight to 0% enables a simple rollback.

We used Datadog to monitor traffic during migration, checking several metrics.

First, we monitored whether traffic rates were as intended. The following image shows traffic rates visualized from the request ratios between Fastly and Cloudflare.

Next, the image below shows the ratio of requests with non-2xx status codes out of all Cloudflare requests. Monitoring these metrics during traffic increases is important.

Since Fastly and Cloudflare exhibit no major visible changes from the client’s perspective, we compared their cache rates, request numbers, and bandwidth usage.

Though not all service migrations had zero incidents, these approaches helped avoid major incidents and minimized impact during incidents.

CDN as a Service

For the next step after migration, we aim for developer self-service, transitioning from centrally managed CDN services by the Network team to "CDN as a Service."

Here, I’ll introduce two initiatives toward "CDN as a Service".

CDN Kit

We named the Terraform module created during the migration process "CDN Kit." By using CDN Kit, developers can easily achieve their goals without needing to define several Terraform resources. The Platform team could provide best practices in one place instead of requiring changes to individual service configuration files.

For example, if the requirement is simple that access the origin via Cloudflare, a developer can use CDN Kit as follows:

module "cdn_kit" {
  source = "..."

  company        = "mercari"
  environment    = "development"
  domain         = "example.mercari.com"

  endpoints = {
    "@" = {
      backend = "example.com"
    }
  }
}

Though simple from a developer’s perspective, using CDN Kit automatically creates various resources. Examples:

Automated logging to BigQuery
- Normally, Cloud Functions are used to log Cloudflare data into BigQuery (document). However creating these for each service is cumbersome, so necessary resources are automatically created with CDN Kit.
Creation of Datadog monitors
Issuance of auto-updated SSL/TLS certificates

Permission Granting System

Cloudflare’s dashboard is a powerful tool for interactive access analysis. However, several challenges needed resolution to make the dashboard accessible to developers:

Managing retired employees
Automating permission grants

For the first challenge, we solved it by enabling SSO on Cloudflare’s dashboard and using Okta as the identity provider (document). Mercari uses Okta, with the IT team managing retiree accounts. Thus, removing retiree accounts from Okta also automatically removes their access to Cloudflare’s dashboard, eliminating the need for direct Network team involvement.

For the second challenge, we created a system that operates in conjunction with our existing internal system. The following is an overview diagram:
※ Team Kit is a Terraform module for managing developer groups.

The Terraform modules for managing developer teams (Team Kit) and managing Cloudflare (CDN Kit) are managed in a GitHub repository. We created a GitHub Actions Workflow to automatically detect module updates. Upon detection, it generates permission management manifest files and commits them to the GitHub repository, as shown below:

account_id: [Cloudflare Account ID]
zone_id: [Cloudflare Zone ID]
zone_name: [Cloudflare Zone Name]
teams:
- team_id: [ID of Team Kit]
  roles:
  - Domain Administrator Read Only
users:
- email: [email address]
  roles:
  - Domain Administrator Read Only

On detecting changes in the manifest files, another GitHub Actions Workflow runs, setting appropriate permissions in Cloudflare based on the manifest files.

We adopt managing Cloudflare permissions declaratively through manifest files instead of directly changing them via GitHub Actions Workflow. This enables returning to the correct state based on the manifest even after manual changes.

The permission granting system allows developers to view the dashboard without requesting access from the Network team. Developers have independently identified and resolved issues using the dashboard, affirming the effectiveness of our "CDN as a Service" initiative.

Conclusion

In this article, I introduced our approach to CDN provider migration and described our initiatives for "CDN as a Service" such as the Terraform module named CDN Kit and permission granting system.

Flutter Forward: Crafting Type-Safe Native Interfaces with Pigeon

Sat, 21 Dec 2024 11:00:42 GMT

This post is for Day 17 of Mercari Advent Calendar 2024, brought to you by @howie.zuo from the Mercari Hallo mobile team.

Introduction

Hello! I’m @howie.zuo, an engineer on the Mercari Hallo mobile team. In this article, I will guide you through the process of generating type-safe native bridges using Pigeon.

Flutter is an incredibly powerful framework. With a vast ecosystem of community-supported plugins, you usually only need to write a minimal amount of native code to create a mobile application. However, finding the right plugin that meets your product’s needs can sometimes be challenging. Even worse, the perfect plugin may have already been deprecated. Therefore, it’s essential to think carefully before adopting a plugin, especially if maintainability and security are critical for your project.

While working on a feature to interact with the calendar app in Mercari Hallo, I discovered that the only suitable plugin I found wasn’t being actively maintained and had poor code quality, as evident from its GitHub repository. As a result, I decided to build the functionality myself.

Note: The code examples in this article are simplified for demonstration purposes. You may need to adjust them for your own codebase. The implementation specifics regarding calendar interactions are not included here, as we’ll focus primarily on Pigeon.

What is Pigeon?

Borrowed the description from here since it describes clearly enough.

Pigeon is a code generator tool used to make communication between Flutter and the host platform type-safe, easier, and faster.

Pigeon removes the necessity to manage strings across multiple platforms and languages. It also improves efficiency over common method channel patterns. Most importantly though, it removes the need to write custom platform channel code, since pigeon generates it for you.

Installation

Start by installing the latest version of Pigeon (22.7.0 as of this writing) in your project’s pubspec.yaml:
```
dev_dependencies:
    pigeon: ^22.7.0
```
Optionally, run dart pub get if your environment doesn’t automatically refresh dependencies.

Configuration

Create a folder named pigeon at the root of your project, and then create a file named message.dart inside the pigeon directory.
```
ROOT_PATH_OF_YOUR_PROJECT/pigeon/message.dart
```
You can choose a different file structure or naming convention if it suits you better.
Import the Pigeon package at the top of your message.dart file:
```
import 'package:pigeon/pigeon.dart'; 
```

Define the input data structures:

class Request {
  Request({
    required this.payload,
    required this.timestamp,
  });
  Payload payload;
  int timestamp;
}

class Payload {
  Payload({
    this.data,
    this.priority = Priority.normal,
  });
  String? data;
  Priority priority;
}

enum Priority {
  high,
  normal,
}

You can find a list of supported data types here. Pigeon also supports custom classes, nested data types, and enums. In Swift, Kotlin, and Dart, you can use sealed classes for a more organized data structure.

Configuration settings

Place the following code at the top of your message.dart file. This tells Pigeon how you want it to generate the code:

@ConfigurePigeon(
  PigeonOptions(
    dartOptions: DartOptions(),
    dartOut: 'lib/pigeon/message.g.dart',
    kotlinOptions: KotlinOptions(
      package: 'com.example.pigeon',
    ),
    kotlinOut:
        'android/app/src/main/kotlin/com/example/pigeon/Message.g.kt',
    swiftOptions: SwiftOptions(),
    swiftOut: 'ios/Runner/Pigion/Message.g.swift',
  ),
)

Pigeon options also support other languages like C, Java, and Objective-C.

Define the output data structures and method interface.

Add the following code at the end of your message.dart file:
```
class Response {
  Response({
    this.result,
  });
  String? result;
}

@HostApi()
abstract class MessageApi {
  bool isAvailable();

  @async
  Response send(Request req);
}
```
The @HostApi() annotation is used for procedures defined on the host platform that can be called by Flutter. Conversely, @FlutterApi() is for procedures defined in Dart that you want to call from the host platform.

The @async annotation indicates that the method is asynchronous.

Code generation

Once the interface is defined, generate the code by running:

flutter pub run pigeon --input pigeon/message.dart

This command will generate code for each platform:

lib/pigeon/message.g.dart
android/app/src/main/kotlin/com/example/pigeon/Message.g.kt
ios/Runner/Pigeon/Message.g.swift

Android Implementation

Create a class named MessageHandler that implements the MessageApi interface:

class MessageHandler : MessageApi {
    fun setUp(message: BinaryMessenger) {
        MessageApi.setUp(message, this)
    }

    override fun isAvailable(): Boolean {
        // your logics go here

        return true
    }

    override fun send(res: Request, callback: (Result<Response>) -> Unit) {
        // get the input
        val data = res.payload.data

        // your logics go here

        // return result asynchronously use callback
        callback(Result.success(Response()))
    }
}

isAvailable and send are the methods we defined earlier. Feel free to implement your own logic inside these methods to handle requests from the Flutter side.

You may have noticed the setUp method; we’ll use this to attach the MessageHandler to the Flutter engine. Override configureFlutterEngine in MainActivity (if it isn’t already present):

class MainActivity: FlutterActivity() {
    override fun configureFlutterEngine(flutterEngine: FlutterEngine) {
        super.configureFlutterEngine(flutterEngine)

        // setup the event handler
        MessageHandler().setUp(flutterEngine.dartExecutor.binaryMessenger)
    }
}

That’s the Android part done. Now let’s move on to the iOS implementation.

iOS Implementation

Similarly to Android, create a class named MessageHandler that implements the MessageApi protocol:

class MessageHandler : MessageApi {
    func setUp(binaryMessenger: FlutterBinaryMessenger) {
        MessagesApiSetup.setUp(binaryMessenger: binaryMessenger, api: self)
    }

    func isAvailable() throws -> Bool {
        // your logics go here

        return true
    }

    func send(res: Request, completion: @escaping (Result<Response, any Error>) -> Void) {
        // get the input
        let data = res.payload.data

        // your logics go here

        // return result asynchronously use callback
        completion(.success(Response()))
    }
}

The class structure is quite similar to the one we created for Android.

Just like in Android, we need to attach MessageHandler to the Flutter engine here as well. Open AppDelegate.swift and insert the following lines inside application(_:didFinishLaunchingWithOptions:):
```
let controller : FlutterViewController = window?.rootViewController as! FlutterViewController
MessageHandler().setUp(binaryMessenger: controller.binaryMessenger)
```

Flutter

Finally, let’s see how to call the host platform methods from Flutter.

For isAvailable

final messageApi = MessageApi();
final isAvailable = messageApi.isAvailable();

For send, which is an asynchronous function:

final messageApi = MessageApi();
final res = await messageApi.send(Request(
  payload: Payload(
    data: 'Hello, Pigeon!',
    priority: Priority.normal,
  ),
  timestamp: DateTime.now().millisecondsSinceEpoch,
));

The code above is straightforward and should be easy to understand.

A few more things

Pigeon also supports macOS, Windows, and Linux.
There are more features not covered in this article that you can explore, such as EventChannelApi.
As a Flutter engineer, you don’t need to be an expert in platform-specific languages, but having some experience in Android or iOS development will undoubtedly be helpful in the development of native API-dependent functionality.

Reference

Some of the resources you may also find useful
https://docs.flutter.dev/platform-integration/platform-channels
https://pub.dev/packages/pigeon
https://github.com/flutter/packages/blob/main/packages/pigeon/example/README.md

Tomorrow’s article will be by @naka. Look forward to it!

Mercari’s Seamless Item Feed Integration: Bridging the Gap Between Systems

Mon, 16 Dec 2024 10:00:30 GMT

Introduction

Hello, I’m @hiramekun, a Backend Engineer at Merpay’s Growth Platform.
This article is part of the Merpay & Mercoin Advent Calendar 2024.

While the Growth Platform is a part of Merpay, we are involved in various initiatives that extend beyond Merpay itself. One such project was the re-architecture of our item feed system. I will introduce the insights we gained from this initiative!

Background

An item feed is a data format and system for managing information from online stores and product catalogs, which is then distributed to various sales channels and advertising platforms. At Mercari, we connect our product data to various shopping feeds so our items can be displayed as ads, which is crucial in promoting products on external media.

For example, Google’s Shopping tab includes listings from numerous sites, including Mercari.

(Source: Shopping tab in Google)

Challenges

Historically, different item feed systems were independently created and managed by various teams, leading to several challenges:

Each system had distinct teams responsible for implementation and maintenance, increasing communication costs.
Although there are common processes, such as retrieving item information and filtering unwanted items, each team implemented them uniquely, resulting in varied issues across systems.
Different systems used different data sources, leading to real-time delays in reflecting item status changes in the feed.

Goals

To address these challenges, we launched a new microservice dedicated to item feeds to provide a unified implementation for all collaborators within a single system. There was also the option of adding features to existing microservices owned by the Growth Platform. However, we decided to launch a new microservice for the following reasons:

To prevent further complicating the roles of existing microservices, which are already extensive.
To minimize the impact on other systems, the design must be adjusted to meet the distinct characteristics of each external service.
Due to the high RPS of item renewal events, scaling according to system demands may be necessary.

Common tasks like filtering configurations, data retrieval, and metadata assignment should be integrated into a single system to ensure that updates are universally applied across services.

While core functionalities are consolidated, it’s crucial to maintain separate implementations for each external service’s unique needs. This separation allows new external services to be integrated with minimal adjustments. Requests made to external APIs must be adaptable to various endpoints and rate limits.

Error handling is also critical. Given the inevitability of encountering external API errors, a retry-capable design is essential to mitigate these potential issues.

Technical Approach

Architecture

The following outlines the architecture. We split processing into workers for common tasks and those specific to linked services (Batch Requesters), connecting them via a Pub/Sub system. This architecture has several benefits:

Allows scaling based on the specific requirements of each worker.
Separates requests to internal microservices from external API requests to isolate unpredictable external API behaviors.
Adding a new batch requester as a subscriber to Pub/Sub can add new external services without altering existing common components.
In case of a surge in item status update events, the Pub/Sub Topic acts as a message queue to enhance system stability.

Let me share each worker in a little more detail.

Common Processing Worker

This worker subscribes to Pub/Sub Topics to receive real-time item status updates from other services. It performs common tasks like adding additional item data, filtering out unsuitable items based on the filter settings, and publishing the processed data to an internal Pub/Sub Topic.

Configured with Horizontal Pod Autoscaler (HPA), this worker dynamically adjusts the number of pods based on CPU usage.

Service-Specific Worker (Batch Requester)

Each batch requester is responsible for subscribing to the Pub/Sub Topic for feed-customized item information for its respective service. Because external API requests must be executed continuously on a second-by-second basis, we implemented these requesters in Go and deployed them as Deployments, not CronJobs. Deployments offer finer control over execution intervals and scalability.

Error handling is also essential. Since requests can fail due to temporary errors in external APIs or network errors, we have implemented a retry feature. This system utilizes the retry mechanism of Pub/Sub and features the following.

The batch requester receives messages from Pub/Sub and stores them in memory as a batch.
At regular intervals, the batch is sent to an external API.
If the submission is successful, the system acknowledges Pub/Sub messages corresponding to all items in the batch.
If the transmission fails, the system negatively acknowledges all corresponding messages and Pub/Sub will resend the message.

Since we want to reflect the status of items in the feed in real time as much as possible if a retry fails a certain number of times, it is forwarded to the Dead-letter topic, and subsequent requests are given priority.

As part of our service level objective (SLO), we monitor the percentage of products correctly reflected in the product feed. We are currently meeting this SLO, so there is no need for a job to retry processing the products accumulated in the Dead-letter topic. However, we might consider developing such a job in the future.

Conclusion

By building this item feed system, we can now distribute items to the feed in near real-time. Separating the common implementation from the specific implementation for each external service has also made it easier to add new services. We plan to add new services and customize feed data.

The next article is by @goro. Please continue to enjoy!

LLMs at Work: Outsourcing Vendor Assessment Toil to AI

Sun, 15 Dec 2024 11:00:22 GMT

This post is for the December 15th installment of Mercari’s Advent Calendar 2024, brought to you by Daniel Wray (Security Management), Simon Giroux (Security Engineering). Banner illustration: Dall-E 3

TL;DR

As Mercari scales, its Security Management Team faces increasing demands for third-party service evaluations. Traditional vendor reviews rely on cumbersome, manual processes (a.k.a toil), which often involves lengthy questionnaires. To streamline this, Mercari is experimenting with employing code and Large Language Models (LLMs) to automate the information-gathering phase, significantly reducing review time. By extracting and analyzing publicly available data, the AI assisted solution provides faster, more consistent assessments while minimizing manual intervention. This approach enhances efficiency, allowing security teams to focus on managing actual risks rather than administrative tasks.

Introduction

Why are we doing these checks in the first place?

Question: Why do companies conduct reviews before authorizing the use of new third party services (i.e. cloud services such as SaaS)?

How this question should be answered, and how deep such checks go, will often depend on the compliance requirements and risk appetite of the organization, but ultimately it boils down to the idea of gaining a sufficient level of confidence, or trust, in the security posture of the external service or vendor, and documenting evidence of the checks performed to reach this conclusion.

Efforts to establish that trust can often explode into a long list of bureaucratic processes, and seemingly endless spreadsheets of compliance checkboxes to tick, in an attempt to ensure consistent and auditable criteria.

Question: Why is it important to establish that trust?

There are very few, if any, companies who can build out all the tooling they need internally; reaching out for external assistance with some part of the business will always be necessary, and doing so involves the need to trust a third party with that work. When businesses work with outside partners, or use external processors or service providers, they gain much-needed support, but also face risks inherent in the use of that specific service.

By the nature of using an external service, whatever internal information the service might handle, such as internal communications, intellectual property, or user data, ends up being stored or processed on someone else’s servers, which opens the door to the potential risk of data leaks from those servers. Moreover, when integrating these external services with other company systems, there’s another layer of risk—if the vendor’s systems are compromised or if a malicious insider is at play, it could lead to a breach that impacts the company’s data and systems beyond the scope of however the external service is being used.

This is where security teams may start to get nervous about third party and supply chain risk.

The Security Management Team at Mercari, which is in charge of reviewing applications to use external services, receives a significant number of such requests per year. As the company continues to grow, this number is sure to increase.

As a team we want to encourage other teams’ innovation and experimentation into new tools and technologies that could improve employee productivity, provide new insights or improve our application’s user experience. However, at the same time we need to balance this against the challenges and risks involved in managing tool sprawl, and find ways to make our security checks scale to these number of requests.

What might this check process look like?

Coming back to the original issue: To consult on the risk associated with onboarding a new external service, and seek approval for implementing it, teams who want to use a service will check with the Security Management Team. The Security Management Team wants to understand the service and evaluate the extent of the risks the tool could entail based on the functionality, use-case, information handled, and how it connects to our environment, and so on.

The assessment process for a new external service might look like this:

Ask for the name of the service
Ask for links to some documentation about the service
Ask the applicant to describe what the service will be used for (i.e. what problem will it solve?)
Ask the applicant to describe what kind of data will be stored or processed by the service
Ask who will be the owner of the service if it is approved and onboarded

Then the Security Management Team would take that information and begin an investigation. The goal is to see if we can trust this external service and its vendor.

Could using this service expose our infrastructure or data to unreasonable risks?
Are the vendor’s security controls sufficient for us to trust them to keep our data safe?
Do the vendor’s controls meet security standards and compliance requirements for the data that they may be responsible for processing for us?
Are there any other potential security risks inherent in the use of this service, or its vendor?
And so on…

While the Security Management Team leads the review process with a focus on information security risk, other teams such as the Privacy Office and Product Security Team may also be involved in the review and approval process depending on the nature of the service, the data it will handle, and how the applicant intends to use it.

Below is a high-level representation of what our process used to look like. While there were numerous issues with this process, including the number of times we had to reach out to the applicant, one of the key issues was the amount of information we had to search for manually on the Internet.

Image 1: Simplified representation of a manually executed vendor assessment process

Legacy and emergent risk assessment tools

The traditional way of conducting an evaluation like this would be to take a spreadsheet with a few pages of questions, send it to the vendor, ask them to fill it in, evaluate their answers, then approve or reject the use of the service—depending on the risks identified, one’s level of risk tolerance, and the necessity of the service. With the back and forth involved in answering and clarifying questions, this process can become quite heavy and take a significant amount of time to complete.

Recently, Trust Centers are emerging as a more modern way to move away from this questionnaire-based approach, and are becoming more common at European and American companies. These pages publicly list compliance standards, laws and regulations that a company claims to follow, often alongside details of their security and privacy controls. An interested party can then request evidence of this compliance directly from the portal (such as certifications or audit reports) and confirm for themselves that the vendor is doing what they are claiming.

Despite the growth in popularity of Trust Centers, they are yet to be universally adopted (even Mercari is yet to publish our own). Without a Trust Center to review, sending the vendor a questionnaire remains the best approach. Even when there is a Trust Center, a company might still choose to send a questionnaire, as it allows the company to ask their own custom set of questions based on their specific risk appetite and points of concern, and may be necessary in order to meet certain regulatory requirements which ask for answers to questions that a Trust Center may not cover. To help vendors answer these questionnaires, some modern governance, risk, and compliance (GRC) tool providers offer AI-assisted functionalities to handle incoming questionnaires. Questions are automatically answered based on a knowledge base of previously-given answers and documentation, with the help of Large Language Models (assuming that the spreadsheet isn’t formatted too artistically for the tool to understand). A requester that also uses a similar GRC tool could then automatically review the answers against their internal questionnaire, and highlight any points that might be missing. These functionalities streamline the process of checking boxes, identifying findings, asking stakeholders to handle them, and finally authorizing (or refusing) the use of a new external service.

GRC Engineering is slowly establishing itself as the obvious next level of evolution. Bringing Agile, DevOps, CI/CD and paved roads in GRC practices should help security teams to better scale with their company. This means having assessments and controls as part of the development process, and providing guidance as early as possible, not just before the release. A precursor idea to this was partially implemented in Google’s Vendor Security Assessment Questionnaires (VSAQ). The questionnaire is in JSON format, allowing the interface to dynamically adapt itself based on the answers, and provide just-in-time guidance when the answer given is already known to be insufficient. This questionnaire also makes it readable by code, removing some of the need to manually interpret answers.

Leveraging LLMs to assess vendors

Sending questionnaires back and forth consumes a lot of time from everyone and can significantly delay the implementation of a service if the check criteria is not clear.

What if we could reduce some of the pain of doing third party risk reviews this way, by creating clearer criteria to highlight the specific areas that a reviewer should focus on, while enabling the auto-collection and analysis of information and evidence on the specific security control requirements we care about?

Internally, we identified a large number of vendors for which, based on the inherent risk of their service, a more lightweight semi-automated approach could be appropriate. For these, the Security Management Team decided to leverage code and Large Language Models to enable us to move fast, and evaluate using clearer and more codified criteria against publicly available information from the vendor, while still appropriately managing risk and maintaining a reasonable level of confidence and trust in the vendor.

Many mature business to business (B2B) vendors already extensively publicize their security practices, which laws and regulations they are subject to, and which compliance standards they have been certified on. Vendors are already openly signaling what level of security and compliance maturity we should be expecting from them. We just have to find a way to read, interpret and understand the endless pages of legalese and jargon in their Privacy Policies, Terms of Service, certificates, White Papers, and Trust Centers.

If successful, this approach could allow us to reduce the need for more time and resource-intensive manual reviews where sufficient information was already publicly available. It would also allow us to focus on those where information could not be obtained, services with a higher inherent risk (e.g. those involving significant system integration or access to large amounts of highly sensitive information), and those requiring additional custom questions or checks for regulatory compliance.

Mercari took inspiration from these emergent approaches, while trying to find a balance that makes sense for us to ensure faster and more efficient review of external services.

Third party website review as code

To be able to learn about the service and its vendor, the risk assessment process requires the analyst to read about the product, understand what it will do, and what information it will store or process. This traditionally involves a lot of searching the internet and reading web pages.

To make this information-gathering easier, the Security Management Team collaborated with the Security Engineering Team, who leveraged open source frameworks, Google’s powerful search engine, and Large Language Models to create a solution.

Supplemented with this automation, the new review process looks like this:

Image 2: Simplified representation of the vendor assessment process for external services

In particular, the introduction of LLMs to this stack is what makes this approach possible. LLMs (we use OpenAI’s GPT-4o in this case, but models that can call tools like Google’s Gemini or Anthropic’s Claude would work too) can read any documentation given to them and provide short answers to any question we might ask.

The challenge is that our review process involves a lot of questions, and follow-up questions based on the answers to these questions, and so on. We can’t simply write a long prompt and hope that the LLM’s answers will tell us everything we want to know and be grounded in reality.

One approach is to use Retrieval Augmented Generation (RAG) to feed documents to a LLM, then ask questions and get answers based specifically on those documents. This is the approach we have taken at Mercari, as it enables us to focus the LLM’s attention on documentation we know is relevant, and reduces the likelihood of both hallucinations and answers based on irrelevant information.

Below is a simplified overview of our approach, which aims to gather the necessary information while minimizing the time and effort required by the applicant, the reviewers, and the vendor.

Image 3: Simplified representation of the role of LLM-powered information gathering in the review process for vendors

It’s time to get hands-on and demonstrate how we can use this automation. For the purposes of this article, we will demonstrate using a fictitious service “PayQuick Cloud Pro”, provided by the fictitious vendor “PaySmooth Solutions”.

The Python code below demonstrates the basic concepts implemented in our AI Agent. First, we take note of the current time. The last code executed in this demonstration contains the total execution time.

import time
start = time.time()

Setting details about the external service and vendor

from llm_code import Profile

profile = Profile(
        **{
            "company": "PaySmooth Solutions", # Enter the company name here
            "product": "PayQuick Cloud Pro", # Enter the product name here
            "url": "https://www.paysmooth.com/payquick", # Enter the product's URL here
        }
    )

Customizing questions

The questions themselves are defined as a function in a Python library. The script sends the ‘profile’ of the external service as a parameter, and a custom questionnaire comes out. This allows us to better control the flow and ask follow-up questions dynamically based on answers received.

Here are some examples of questions for demonstration.

from questions_code import prepare_questions
from IPython.display import Image, display, Markdown

questions = prepare_questions(profile)
for i, question in enumerate(questions):
    if i > 2:
        break
    display(Markdown(f"## Question {i+1}: {question.get('label', 'General')}"))
    for key in question.keys():
        display(Markdown(f"**({key})**n{question[key]}n"))

Question 1: General

(goal) The team performing the assessment isn't necessarily aware of what this service is doing. This question will tell them what the product is supposed to do, how it is supposed to be used, and what kind of data it is supposed to process.
(main) What is the purpose of ‘PayQuick Cloud Pro’ by PaySmooth Solutions? Which problem is it promising to solve? Why would a customer consider using it?
(expected) A brief description

Question 2: General
(goal) A service can be used by different types of users, such as administrators, end-users, or developers. This question will help the team understand who is the target market, operators, and users of the service.
(main) Who is the target market, operators and users of PaySmooth Solutions PayQuick Cloud Pro?
(expected) A brief description

Question 3: General
(goal) The team needs to understand the key features of the service to assess the risks associated with it. This question will help the team understand what the service is supposed to do.
(main) What are the key features of PaySmooth Solutions PayQuick Cloud Pro?
(expected) A list of features

Using Langgraph to configure an AI agent

The Langgraph library provides a nice framework to control the execution flow of an AI agent. This agent can then use tools to perform some of the tasks and use a LLM to produce the final response to a question.

As described by the graph below, the agent:

receives the question from the script,
decides if it needs to use Google Search to find relevant documents,
gives back the content recovered to the LLM to decide what to do with it,
will search the internet again if content isn’t good, or will give up if there were too many attempts,
asks the LLM to answer the question.

from llm_code import build_graph
from langchain_core.runnables.graph import CurveStyle, MermaidDrawMethod, NodeStyles

graph = build_graph()
display(
    Image(
       graph.get_graph().draw_mermaid_png(
            draw_method=MermaidDrawMethod.API,
        )
    )
)

Asking the agent to answer each question

With the agent defined, we can then pass all our questions and ask it to search for answers.

from llm_code import perform_assessment
answers = perform_assessment(questions, profile, graph)

Searching the internet for answers about PaySmooth Solutions - PayQuick Cloud Pro
* Q. What is the purpose of ‘PayQuick Cloud Pro’ by PaySmooth Solutions? Which problem is it promising to solve? Why would a customer consider using it?
* Q. Who is the target market, operators and users of PaySmooth Solutions PayQuick Cloud Pro?
* Q. What are the key features of PaySmooth Solutions PayQuick Cloud Pro?
    ! truncated to 7456 tokens
* Q. What category of product is PaySmooth Solutions PayQuick Cloud Pro in?
* Q. What is the list of companies or customers who are using PaySmooth Solutions PayQuick Cloud Pro?
* Q. According to the Trust Center page, or the official site, what laws and regulations is PaySmooth Solutions PayQuick Cloud Pro compliant with?
    ! truncated to 9832 tokens
    ! truncated to 9832 tokens
* Q. According to the Trust Center page, or the official site, what compliance standards is PaySmooth Solutions PayQuick Cloud Pro following?
* Q. According to the Trust Center page, or the official site, what security standards is PaySmooth Solutions PayQuick Cloud Pro compliant with?
    ! truncated to 7334 tokens
    ! truncated to 7334 tokens

After asking all questions and follow-up questions, answers are returned in JSON format, which allows us to easily manipulate them.

Producing the report

With the answers collected, we can ask the LLM to produce an executive summary and a detailed report.

from llm_code import ask_llm
from prompt_code import make_summary_prompt
from reporting_code import summary_markdown, report_markdown

summary_prompt = make_summary_prompt(answers, profile)
summary = ask_llm(summary_prompt)
report = report_markdown(answers, profile)

display(Markdown(summary_markdown(summary, profile)))

Executive Summary Report
- Company: PaySmooth Solutions
- Product: PayQuick Cloud Pro
- URL: `https://www.paysmooth.com/`
- Date: 2024-11-10

Goal of the product, why are we deploying it, how will it help us solve issues we are facing?
- Goal: Streamline payment processes, enhance security, and support business growth.
- Deployment Reason: Manage multiple payment methods efficiently.
- Solution: Secure transaction processing and financial services support.

What are the laws and regulations that this product is compliant with?
- Specific laws and regulations are not clearly listed, but it complies with ISO27001, PCI DSS, and Privacy Mark.

What are the compliance standards that this product is compliant with?
- ISO/IEC 27001: Information security management.
- PCI DSS: Credit card industry security standard.
- Privacy Mark: Personal information protection standard in Japan.

What are the security standards that the company is following?
- ISO/IEC 27001
- PCI DSS
- Privacy Mark

What kind of data this service is meant to process or store?
- Payment data, including credit card information, digital wallet transactions, and bank transfers.

Are there risks that were highlighted that the Risk and Security team should be made aware of?
- Risk: Potential for data breaches or fraud.
- Impact: Financial loss, reputational damage, and regulatory penalties.

Are there any countermeasures that should be implemented to mitigate risks of using this service?
- Implement robust security measures like EMV 3D Secure and regular vulnerability assessments.
- Ensure compliance with PCI DSS and ISO/IEC 27001 standards.
- Conduct regular security audits and employee training.

display(Markdown(report))

Report for PaySmooth Solutions PayQuick Cloud Pro (2024-12-15)
URL: https://www.paysmooth.com/payquick

Answers
1 (General) What is the purpose of ‘PayQuick Cloud Pro’ by PaySmooth Solutions? Which problem is it promising to solve? Why would a customer consider using it?

Answer (100.0% confidence): PaySmooth Solutions PayQuick Cloud Pro provides comprehensive online payment services, offering a wide range of payment methods including credit cards, carrier payments, and various digital wallets like PayPay, AmazonPay, and ApplePay. It aims to solve the problem of managing multiple payment methods for businesses, ensuring secure and efficient transaction processing. Customers would consider using it to streamline their payment processes, enhance security with measures like EMV 3D Secure, and support business growth through financial services and consulting.

… snip …

6 (Compliance) According to the Trust Center page, or the official site, what laws and regulations is PaySmooth Solutions PayQuick Cloud Pro compliant with?

Answer (0.0% confidence): The specific list of laws and regulations that PaySmooth Solutions PayQuick Cloud Pro is compliant with is not clearly found on the official website or related pages. The site mentions compliance with ISO27001, PCI DSS, and the Privacy Mark, but does not provide a detailed list of specific laws and regulations such as GDPR, CCPA, APPI, etc.

7 (Compliance) According to the Trust Center page, or the official site, what compliance standards is PaySmooth Solutions PayQuick Cloud Pro following?

Answer (100.0% confidence): PaySmooth Solutions PayQuick Cloud Pro follows the following compliance standards:
1. ISO/IEC 27001: This is a global standard for information security management, and PaySmooth Solutions PayQuick Cloud Pro has obtained conformity certification for all of its business sites.
2. PCI DSS: PaySmooth Solutions PayQuick Cloud Pro's services are fully compliant with PCI DSS version 3.2.1, which is a global security standard for the credit card industry.
3. Privacy Mark: This certification indicates compliance with the Japanese Industrial Standard for personal information protection, JIS Q15001:2017.

… snip …

Reviewing the report

The Security Management Team (and any other teams involved in the review for the service) will then evaluate the reports to quickly gain a broad understanding of the service to guide their decision-making. To use their time as efficiently as possible, in most cases, they will read just the Executive Summary and only refer to the more detailed report if needed to confirm any specific concerns.

Following a simple manual and based on established and defined criteria, the team will then carry out their review. In some cases, such as where there isn’t much information available about the service online, the team may then decide to perform a deeper analysis (and perhaps bring out the spreadsheets), but in most cases, particularly for services and vendors with a high level of compliance maturity, the information from the application form and the LLM’s report should be enough to determine whether (or not) the service meets all our basic requirements and the information security risk is at an acceptable level, and if so, give their blessing by approving the service and adding it to our List of Approved External Services (with appropriate restrictions on how it may be used).

We can grasp whether sufficient information was available online to answer each question based on the ‘confidence score’ that the LLM assigns to each of its answers. If the confidence score is low, there was likely little information available. If the score is zero, there was nothing that the LLM thought it could use.

If there are many low-or-zero confidence scores in the report, we can disregard the report and resort to the old-fashioned method of sending a questionnaire to the vendor, but if there are just a few, we can reach out to the vendor and simply ask them these few specific questions; we may have an answer for this in just hours, or minutes during a call, rather than the weeks (or longer) it typically takes to complete a full questionnaire.

from reporting_code import report_confidence
confidence_report, improvements = report_confidence(answers, profile)

display(Markdown(confidence_report))

Confidence Report
- Percentage of answers collected from the vendor's web pages: 100.0%
- Average confidence score: 62.5%
- Number of answers with low confidence scores: 2

Answers with low confidence scores:
- (0% confidence) What is the list of companies or customers who are using PaySmooth Solutions PayQuick Cloud Pro?
- (0% confidence) According to the trust center page, or the official site, what laws and regulations is PaySmooth Solutions PayQuick Cloud Pro compliant with?

Some questions might fail, especially if the web site isn’t friendly with automation, because the information isn’t where we expect to find it, or because the context window wasn’t big enough to read all pages. For these questions, a manual check is likely to be necessary. We could also ask the vendor to improve their pages to cover these questions. See below for more about this.

How much does executing this script cost?

Performing a manual assessment can take several hours, and the results are likely going to be inconsistent. Let’s say that each assessment takes a total of six hours to complete (total people-hours spent by the applicant and all reviewers) and assume (for ease of calculation, not based on actual figures) that the average salary of those involved in the review is 10 million yen per year (equivalent to roughly 5000 yen per hour). Each review would then cost on average 30,000 yen, mostly spent searching the internet, reading web pages, and collating information into a report. If we were to do 250 reviews per year, this would represent an annual cost of around 7.5 million yen.

Using automation and LLMs can greatly reduce this time spent searching the internet looking for answers, as well as the time spent writing down every detail along the way and summarizing it in a report at the end.

from reporting_code import calculate_token_counts, token_count_markdown

token_report = calculate_token_counts(profile)
display(Markdown(token_count_markdown(token_report)))

Token Usage Report
- Total costs: 1.29$

Model: gpt-4o-2024-08-06
- Total calls: 32
- Total tokens: 248369 (1.29$)
- Input tokens: 243518 (1.22$)
- Output tokens: 4851 (0.07$)

In this example, we asked just 8 questions for a total of $1.29, but in a normal assessment of 32 basic questions plus follow-up, which can involve up to 100 total questions, the actual token cost is closer to $10.

If running this report reduces the people-hours required for a review by just 25%, this translates to a hypothetical saving of 7500 yen (~$50) in personnel costs, for a return-on-investment of 500%.

It’s not just the financial benefit—by streamlining the process and reducing the people-hours required to carry out the review, we reduce the length of the period the applicant has to wait for their external service to be approved. This helps the business to move faster. It is clear that using automation to conduct the initial assessment helps significantly.

Asking the vendor to provide additional details on their website

We are now done with our assessment. This was a one way process; our script searched the internet and collected answers to the questions we were interested in. Bonus—the vendor didn’t have to do anything—assuming all the information we needed was already published somewhere on their website.

But what if not all the information we needed was on their website? For information that is necessary for us to move forward, we will have to reach out to the vendor. One day, security teams across companies might talk to each other through APIs and secure handshakes. In the meantime, we could also let the vendor know what we couldn’t find by signaling them through their corporate web site.

The following step lists the questions for which our agent couldn’t find answers and performs a GET request on [vendor.domain]/compliance.txt for each one with the question as a parameter.

Unlike robots.txt or security.txt, compliance.txt isn’t used as a standard (to this date). The query is likely to fail. However, a vendor that monitors for errors on their corporate web site is likely to notice the hits on /compliance.txt and see the question. The user-agent configured to perform this request points back to this blog post. The compliance.txt file can actually be empty, especially if everything is already documented in the webpages. For example, the file could contain the URL to the vendor’s Privacy Policy and any statements of evidence regarding their compliance. If these pages are hard to process through automation (Javascript), populating this file in plain text with terms of services, privacy policies, and other details about the company’s compliance status directly in this file could actually simplify the overall review process. Protecting the agent against prompt injection attacks is however important.

from reporting_code import request_for_improvement

for answer in improvements:
    request_for_improvement(answer, profile)

Requesting `https://paysmooth.com/compliance.txt?question=What+is+the+list+of+companies+or+customers+who+are+using+PaySmooth+Solutions+Payment+Gateway%3F`
Requesting `https://paysmooth.com/compliance.txt?question=According+to+the+trust+center+page%2C+or+the+official+site%2C+what+laws+and+regulations+is+PaySmooth+Solutions+Payment+Gateway+compliant+with%3F`

Certified doesn’t mean secure

“How come we were hacked? We are ISO 27001 compliant!” – Some CEO somewhere…

Wait, all you’ve done is demonstrate that you could use an AI agent to read the internet. This is not proving that a vendor is secure!

Indeed, no matter what is written on a vendor’s website—what standards they claim to be compliant with, what certificates and audit reports they are willing to share, what security controls they claim to have implemented—it doesn’t ‘prove’ that the service or its vendor are secure or trustworthy. Performing an assessment isn’t about proving that a company or service is secure, that would require our security engineers to thoroughly assess the vendor’s technical environment, which given a lack of infinite time and resources would not be practical or realistic considering the number of applications we get per year. Even this wouldn’t be enough to say we’ve “proven” anything, and would purely be a point-in-time check at best (not to mention the fact that most vendors would never agree to the burden of being assessed in such a heavy way by us in the first place). At some point, we have to decide how much time and effort should be invested to review an external service for us to trust it enough to use it – to allow it to store or process the information that the applicant wants to use it for, to integrate with whatever other systems they want it to integrate with, or to be part of whatever (potentially critical or user-facing) operation it will be used for.

Which brings us back to what third party risk management actually is and the role of certification against standards in it. The expectations are that a vendor will not claim to be compliant to standards if they are not confident that they put in the work and actually achieved compliance. Even if we were to ask a vendor to fill out a security checklist for us, the trustworthiness of their answers wouldn’t be any different to what they have written or would write on their website.

The vendor’s compliance team already spent a significant amount of time sharing details about their internal practices on their website. The greatest service we can do for them is trusting that information. The second greatest service we can do for them is to only request information from them that we actually need, and that isn’t already available on their website.

Once all teams involved in the review have given the thumbs-up, the ticket is approved, the service added to our List of Approved External Services, and the applicant informed that they are good-to-go (and given relevant advice and warning on using and managing the service securely). This leaves the Security Management Team to move on to follow-up tasks, such as:

Registering the service in our Information Asset Register, along with the data it will store (and process)
Ensuring that any integrations between the service and other company systems is done securely
Ensuring that the new service is integrated to our internal access provider for Single Sign-On
Ensuring that logging and backups are configured appropriately for the system, in line with our policies
Working with our Threat Detection and Response Team to ensure that appropriate monitoring is in place for the new service, particularly if it is expected to handle a critical function or handle highly sensitive information

By simplifying the review process and keeping it toil-free, we also help free up time and maintain the momentum and energy of our Security Management Team to focus on these important next steps, that otherwise may be delayed or fall through the cracks.

Releasing the time spent on the review process allows us to invest it where it can be used more effectively: addressing and treating the risks associated with actually using the new service in our environment.

Conclusion

Using a variant of the script above, together with numerous other improvements to our review process and decision-making criteria, the Security Management Team was able to reduce the average total amount of people-hours necessary to review an external service by approximately 50%. Furthermore, our new process produced multiple other benefits:

The reviewer’s overall understanding of the service increased
Our assessments are now more thorough and consistent
Less mature companies can be easily identified (due to the lack of publicly available information)
The average time from application to approval (during which the applicant can’t use the service) has greatly reduced
Reviewer morale has improved since the process is less demanding and involves less manual, tedious work

Because we are using an LLM to read human-readable pages, there is no need to establish yet another documentation standard to report compliance (as opposed to a yaml or JSON with question IDs, tags, titles, description, etc). This script can request for additional details through a hit on compliance.txt, but isn’t waiting for an answer. By doing so, we simply hope that vendors can update their websites and/or Trust Centers to provide these additional details for the benefit of those looking for the same information that we were.

For us, using automation to conduct the part of our external service review doesn’t totally remove the burden of assessing our vendors, but does liberate time so our team could focus on other important tasks.

Where do we go from here?

Generative AI technologies are evolving quickly. Between the time we wrote this article and the time we published it, Google announced the release of Gemini 2.0 and Project Mariner. Anthropic also recently released Computer Use which would also allow an AI agent to take control of one’s computer. The automation we developed runs in a GCP Cloud Run instance, but nothing would stop someone from running it as a Chrome extension augmented by a LLM, where this LLM would take over the browser and execute a given list of research tasks. One thing is certain: there is huge potential for reducing toil in daily operation work.

— EOF —

print(f"Total execution time: {time.time() - start:0.2f} seconds")

Total execution time: 59.06 seconds

Installation instructions

If you wish to try this notebook, the source code is available here: https://github.com/cerebraljam/llms-at-work

This notebook was developed using Python 3.11 and Visual Studio Code.

From Airflow to Argo Workflows and dbt Python models

Sun, 15 Dec 2024 10:00:12 GMT

This post is Merpay & Mercoin Advent Calendar 2024, brought to you by @Yani from the Merpay Data Management team.

This article describes the journey of Merpay when migrating from Airflow to Argo Workflows and dbt, and the considerations that went into this choice. We will start with an introduction of each tool and the migration criteria that were evaluated, followed by a note to clarify some important terminology. Finally we will close with a blueprint for such a migration, rounding up best practices and common pitfalls we gathered through our own experience.

Tool introduction

Apache Airflow

Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. Its main strength lies in its ability to define workflows as code, allowing for dynamic pipeline generation, testing, and versioning. It also supports a wide range of operators for tasks, further enhancing its flexibility.

Argo Workflows

Argo Workflows is an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes. It supports workflow templates allowing users to define reusable workflow steps and to orchestrate complex jobs that require parallel execution and conditional branching.

dbt

dbt (Data Build Tool) is a data transformation tool that can be used to collaborate on data models. Users can modularize their SQL queries, test and document them before deploying them to production, with auto-generated data lineage which simplifies impact analysis and debugging. dbt compiles and runs queries against specific data warehouses such as BigQuery on Google Cloud Platform (GCP).

dbt SQL models are representations of tables or views. Models read in dbt sources or other models, apply a series of transformations, and return transformed datasets in the form of a SQL SELECT statement. dbt arranges models in a dependency graph and ensures that upstream models are executed before downstream models.

dbt Python models can help you solve use cases that can’t be solved with SQL. They have all the same capabilities around testing, documentation, and lineage. On GCP, Python models are executed via Dataproc which is using PySpark as the processing framework. PySpark is an expressive and flexible API compatible with other popular libraries (e.g. pandas, numpy, scikit-learn, etc).

Migration Criteria

Airflow to Argo Workflows for workflow orchestration

Architecture
Apache Airflow operates as a standalone application, this means that managing resources and scaling can be more of a challenge with Airflow.
Argo Workflows is Kubernetes-native, meaning it’s designed to run on a Kubernetes cluster, which allows for easier scaling and resource management.
Workflow Design
Apache Airflow excels in its ability to define workflows as code, which allows for dynamic pipeline generation, versioning, and testing.
Argo Workflows supports complex workflows with loops, recursion, and conditional logic. Workflows are configured with the native language of Kubernetes: YAML. There is a Python software development kit (SDK) called Hera, which can streamline code with less boilerplate and features like code completion.
Scheduling
Apache Airflow uses its own scheduler, which means that the performance and reliability of the scheduler are dependent on the resources of the machine where Airflow is installed.
Argo Workflows uses Kubernetes CronJob to schedule workflows, leveraging the power of Kubernetes for resource management and reliability.
User Interface
Apache Airflow offers a robust and interactive user interface (UI) which allows users to monitor workflows in real-time, view logs, and even rerun tasks directly from the interface, thus supporting quick and easy debugging.
Argo Workflows provides a straightforward and clean interface for viewing and managing workflows. It may not be as feature-rich as Airflow’s UI, but there is a lot of active development around it.
Community and Support
Apache Airflow has been around for a longer time, it has a larger community and more extensive documentation.
Argo Workflows has a rapidly growing user base with a very active community, and the documentation is improving and expanding rapidly.

In the table below, the characteristics of each tool are evaluated as positive or negative in the context of Merpay’s requirements and overall environment.

	Apache Airflow	Argo Workflows
Architecture	−	+
Workflow Design	+	−
Scheduling	−	+
User Interface	+	+
Community and Support	+	+

Airflow to dbt for task definition

Purpose
Apache Airflow can be used to define complete ETL workflows as well as workflows with arbitrary scripted tasks interacting with a variety of systems.
dbt focuses only on data transformations and modeling, while interacting with a single data warehouse or database.
Dependency Management
Apache Airflow supports dependency management through explicit task and workflow dependencies.
dbt offers built-in dependency management by automatically building dependency graphs based on the connectivity between models, and ensures that transformations are executed in the correct sequence.
Language
Apache Airflow was developed in Python so it offers the full flexibility of the language.
dbt is mainly SQL-based and has secondary support for defining models in Python.
Learning Curve
Apache Airflow can be more daunting for users without prior experience in Python or understanding of basic Airflow-specific concepts.
dbt reduces the learning curve by allowing users to define transformations in a common language like SQL and to manage boilerplate logic (such as materializations) through simple configuration parameters.

dbt has been the tool of choice for SQL-based data transformations in Merpay for a while, so after the migration from Airflow to Argo Workflows, we wanted to explore the feasibility of using dbt Python models for some of our workflows.

Terminology note

Directed Acyclic Graph (DAG)
In Airflow, DAGs represent a collection of tasks, where each node is a task and each edge is a dependency between two tasks.
In Spark, a DAG represents a logical execution plan of computation, where each node is a transformation and the edges show the flow between computations.
Task
In Airflow, a task is the basic unit of work and parallelism, it performs a specific action and it can have upstream and downstream dependencies.
In Spark, a task is also the unit of work but it exists within the broader context of a Spark job. Jobs are represented by DAGs, and are split into stages which are ultimately collections of tasks.
However, in Spark the unit of parallelism is a partition, which is a logical chunk of data in an Resilient Distributed Dataset (RDD). Each partition is processed independently by a single task, performing the same computation on that specific chuck of data.

The key distinction is that Spark’s parallelism is rooted in data partitioning, whereas Airflow’s parallelism revolves around task orchestration. On the surface this might seem as a slight difference but in reality it can have big implications.

Migration process

Initially, our migration involved mostly workflows that cataloged company-wide data gathered from BigQuery’s Information Schema views, Data Catalog and other GCP APIs.
The migration process was for the most part straightforward, but we gathered a few points to act as best practices, as well as a couple of common pitfalls.

Best practices

Express your starting data as Dataframes
Chain preparatory transformations
- A good way to keep things clean is the transform() function
Repartition based on the target API’s quota and other limitations
- Useful functions when managing partitions include agg(), groupBy(), keyBy() and partitionBy()
Interact with APIs on a partition-level
- Mostly use flatMap() or mapPartitions()
Manage the output schema explicitly

Common pitfalls

RDDs fail as a whole
- In contrast to Airflow tasks, it’s harder to gather partial results
Incremental tables are not supported by Python models
- Use a Python model for the latest table and a SQL model for the incremental

Example code

The following example is the complete code for a Python model used for collecting information about all GCP projects, and augmenting that with a column for BigQuery-enabled projects.

from time import sleep
from typing import Iterator

import google.auth
from google.auth.impersonated_credentials import Credentials
from googleapiclient import discovery
from googleapiclient.errors import HttpError
from pyspark.sql import DataFrame, SparkSession
from pyspark.sql.functions import length, col, lit, to_timestamp
from pyspark.sql.types import StructType, StructField, StringType

def model(dbt, session: SparkSession) -> DataFrame:
    dbt.config(materialized="table")
    service_account = dbt.config.get("service_account")

    all_projects = get_projects_from_resource_manager(session, service_account)
    bigquery_projects = get_projects_from_bigquery(session, service_account)

    return (
        all_projects
        .transform(exclude_temp_projects)
        .transform(
            add_bigquery_enabled_column,
            bigquery_projects=bigquery_projects
        )
        .transform(finalize_dataframe)
    )

def get_projects_from_resource_manager(
        session: SparkSession,
        target_principal: str
) -> DataFrame:
    projects = list_projects_from_resource_manager(target_principal)

    schema = StructType([
        StructField("projectId", StringType()),
        StructField("projectNumber", StringType()),
        StructField("lifecycleState", StringType()),
        StructField("labels", StructType([
            StructField("data_bank_card_info", StringType()),
            StructField("data_credit_card_info", StringType()),
            StructField("data_personal_identifiable_info", StringType()),
            StructField("service_corporation", StringType()),
            StructField("service_country", StringType()),
        ])),
        StructField("parent", StructType([
            StructField("type", StringType()),
            StructField("id", StringType()),
        ])),
        StructField("createTime", StringType()),
    ])

    return session.createDataFrame(projects, schema)

def get_projects_from_bigquery(
        session: SparkSession,
        target_principal: str
) -> DataFrame:
    projects = list_projects_from_bigquery(target_principal)

    return session.createDataFrame(projects)

def exclude_temp_projects(projects: DataFrame) -> DataFrame:
    project = col("projectId")

    return projects.where(~(
        project.startswith("sys-")
        & (length(project) == 30)
        & (project.substr(5, 26).rlike(r"(\d+)"))
    ))

def add_bigquery_enabled_column(
        all_projects: DataFrame,
        bigquery_projects: DataFrame
) -> DataFrame:
    return (
        all_projects
        .join(
            bigquery_projects.withColumn("bigqueryEnabled", lit(True)),
            "projectId",
            "left_outer"
        )
        .fillna(False, "bigqueryEnabled")
    )

def finalize_dataframe(df: DataFrame) -> DataFrame:
    return (
        df
        .withColumn("createTime", to_timestamp("createTime"))
    )

def list_projects_from_resource_manager(target_principal: str) -> Iterator[dict]:
    credentials = get_impersonated_credentials(target_principal)
    service = discovery.build(
        "cloudresourcemanager",
        "v1",
        credentials=credentials,
        cache_discovery=False
    )

    request = service.projects().list()

    while request is not None:
        response = request.execute()

        for project in response.get("projects", []):
            yield project

        request = service.projects().list_next(
            previous_request=request,
            previous_response=response
        )

def list_projects_from_bigquery(target_principal: str) -> Iterator[dict]:
    credentials = get_impersonated_credentials(target_principal)
    service = discovery.build(
        "bigquery",
        "v2",
        credentials=credentials,
        cache_discovery=False
    )

    request = service.projects().list()

    while request is not None:
        try:
            response = request.execute()
        except HttpError as e:
            if 403 == e.status_code and "Quota exceeded" in e.reason:
                print(f"Error while listing projects: {e.reason}")
                sleep(1)
                continue
            else:
                raise e

        for project in response.get("projects", []):
            yield {"projectId": project["projectReference"]["projectId"]}

        request = service.projects().list_next(
            previous_request=request,
            previous_response=response
        )

def get_impersonated_credentials(target_principal: str) -> Credentials:
    scopes = ("https://www.googleapis.com/auth/cloud-platform",)
    source_credentials, _ = google.auth.default(scopes)
    return Credentials(source_credentials, target_principal, scopes)

Conclusion

In this article we discussed the criteria that led us to migrate from Apache Airflow to Argo Workflows and dbt Python models. More importantly, we pointed out some key differences regarding the units of work and parallelism between these tools, and laid out a blueprint for such a migration with our best practices and the common pitfalls we observed.

We hope this helps your own journey and see you for the next article tomorrow!

Learnings About Swift Testing

Sat, 14 Dec 2024 10:00:29 GMT

This post is Merpay & Mercoin Advent Calendar 2024, brought to you by @cyan from the Mercoin iOS team.

Hi! My name is Cyan, and I’m one of the members of the Mercoin iOS Team. This will be my first time writing a blog for Mercari, so I am hoping that you’ll enjoy reading this post. In this blog post, I’d like to share some learnings about Swift Testing.

Personally, I think that Swift Testing is easier to use and more complete than XCTest.

Swift Testing is a new Unit Testing framework introduced by Apple at this year’s WWDC24. This is meant to be the successor of the much used XCTest framework. Swift Testing can only be used from Xcode 16, so if your team haven’t updated the project yet, maybe it’s time to update now 🙂

Let’s start!

Attributes and Macros

@Test

When we were using XCTest, we would add test at the beginning of the function name to make the function as a test case.

import XCTest
func test_defaultValue() {
    // ...
}

But for Swift Testing, we don’t need to add test but instead use a @Test attribute.

import Testing
@Test func defaultValue() {
    // ...
}

Same with XCTest’s test functions, we could still add async, throws, and @MainActor on our tests.

#expect

This macro is used for actually doing the checking. It is the same with XCTest’s XCAssert functions. Although, the one key difference of Swift Testing with XCTest is that we don’t need specific functions for different cases on checking.

For XCTest, we can use all of these functions:

XCTAssert, XCTAssertTrue, XCTAssertFalse
XCTAssertNil, XCTAssertNotNil
XCTAssertEqual, XCTAssertNotEqual
XCTAssertIdentical, XCTAssertNotIdentical
XCTAssertGreaterThan, XCTAssertGreaterThanOrEqual
XCTAssertLessThan, XCTAssertLessThanOrEqual

However, in Swift Testing, you can just do it just like these:

#expect(amount == 5000)
#expect(user.name == "Hoge")
#expect(!array.isEmpty)
#expect(numbers.contains(1))
#expect(paymentAmount > 0)

We only need to pass an expression to #expect, which is way simpler and easier to remember.

#require

This macro is used when you want to have a required expectation. Meaning, when this test case fails, the entire test will stop and fail.

try #require(date.isValid)  // ← if it fails here...

#expect(date, Date(timeIntervalSince1970: 0))  // ← then this is not executed

Additionally, this can also be used when you want to unwrap optional values, and stop the test when the said optional value is nil.

let method = try #require(paymentMethods.first)  // ← if .first is nil...

#expect(paymentMethods.isCreditCard)  // ← then this is not executed

Traits

These are new in Swift Testing, and these provide much easier ways to customize our unit tests. There are lots of traits introduced, so I tried to categorize them into 3 categories so it’s easier to remember them:

Detail-related Traits
Condition-related Traits
Behavior-related Traits

Detail-related Traits

Display Name

This trait allows us to add a name to our test case. Of course, we could know from the function name what the test case does, but it would be easier to understand it if we use this Display Name trait since we could add spaces on it.

@Test("Check default value when there’s a plan")
func defaultValueWithPlan() {
    let dependency = Dependency(plan: 1000)
    #expect(selectedAmount == 1000)
}

Trait .bug

This trait allows us to link an issue if the said test case was added after fixing a particular bug.

@Test(.bug("example.com/issues/123", "Check default value when there’s no plan") 
func defaultValueWithNoPlan() throws {
    …
    let firstAmountOption = try #require(amounts.first)
    #expect(selectedAmount == firstAmountOption)
}

Trait .tags

This trait allows us to add a tag to the test case, and be able to see it on the left side panel of Xcode for easier organization of test cases. Firstly, we’d have to have an extension for Tag to add our desired tags.

extension Tag {
    @Tag static var formatting: Self
    @Tag static var location: Self
    @Tag static var playback: Self
    @Tag static var reviews: Self
    @Tag static var users: Self
}

And then, you could use it like this:

struct SwiftTestingDemoTests {
    @Test(.tags(.formatting)) func rating() async throws {
        // add #expect here
    }
    …
    @Test(.tags(.location)) func getLocation() async throws {
        // add #expect here
    }
    …
    @Test(.tags(.reviews)) func addReviews() async throws {
        // add #expect here
    }
}

You’ll see something like this:

You can group tests into a Suite, and then add a tag on that Suite. That would add the tags to all the tests inside that Suite.

@Suite(.tags(.defaultValue))  // ← add .tags here
struct SelectedAmountDefaultValue {
    @Test func defaultValueWithPlan() async throws {
        …
    }

    @Test func defaultValueWithNoPlan() async throws {
        …
    }
}

I’ll share more about Suites later 🙇

Condition-related Traits

Trait .enabled

This trait allows us to specify a condition if we want to run our test case or not.

@Test(.enabled(if: FeatureFlag.isAccordionEnabled))
func defaultValueAccordionState() {
    // ...
}

Trait .disabled

This trait allows us to unconditionally disable a test. This could be useful when you have flaky tests in your project and it is causing delays.

@Test(.disabled("Due to flakiness"))
func flakyTestExample() {
    // ...
}

Trait @available

This trait allows us to add a condition if the test should be run or not depending on the OS version.

@Test
@available(macOS 15, *)
func caseForFunctionThatUsesNewAPIs() {
    // ...
}

Tip:

It is recommended by Apple to use @available instead of checking at runtime using #available.

// ✖︎ Avoid checking availability at runtime using #available
@Test func caseForFunctionThatUsesNewAPIs() {
    guard #available(macOS 15, *) else { return }

    // ...
}

// ⚪︎ Prefer @available attribute on test function
@Test
@available(macOS 15, *)
func caseForFunctionThatUsesNewAPIs() {
    // ...
}

Behaviour-related Traits

Trait .timeLimit

This trait allows us to add a time limit to a test case. It could be useful in a case wherein you don’t want a particular function to run above a certain time threshold.

@Test(.timeLimit(.minutes(5)))
func someMethod() {
    // ...
}

Trait .serialized

This trait allows us to have tests in a Suite to be run in order, instead of all at the same time.

@Suite(.serialized)
struct SelectedAmountDefaultValue {
  @Test func defaultValueWithPlan() {
      ...
  }
  @Test func defaultValueWithNoPlan() {
      ...
  }
}

Now that we’ve discussed Traits, let’s proceed to some tips and tricks that we could use with Swift Testing.

Pairing Traits

You could also use multiple Traits in one test case.

@Test(
  .disabled("Due to a crash"),
  .bug("example.org/bugs/123", "Crashes at <symbol>")
)
func testExample() {
    // ...
}

Suites

You might have noticed that Suites were mentioned a few times in this article. Basically, a Suite is a group of test functions.

annotated using @Suite.
could have stored instance properties.
could also use init and deinit for set-up and tear-down logic, respectively.
initialized once per instance of @Test function.

@Suite(.tags(.defaultValue))
struct SelectedAmountDefaultValueNilPlanTests {
    let dependency = Dependency(plan: nil)

    init() throws {
        ...
    }

    deinit {
        ...
    }

    @Test("Check when there’s initial amount")
    func withInitialAmount() {
        // #expect…
    }

    @Test("Check when there’s no initial amount")
    func withNoInitialAmount() {
        // #expect…
    }
}

Parameterized Testing

When you have some repetitive tests, you could use a parameterized @Test function. An example of repetitive tests would be something like below:

// ✖︎ not recommended
struct CryptoCurrencyTests {

    @Test func includesBTC() async throws {
        let data = try await GetData()
        let currency = try #require(data.first(where: { $0 == "BTC" } ))
        #expect(currency == “BTC”)
    }

    @Test func includesETH() async throws {
        let data = try await GetData()
        let currency = try #require(data.first(where: { $0 == "ETH" } ))
        #expect(currency == “ETH”)
    }

    // ...and more, similar test functions
}

Sure, you could use a for…in loop to repeat a test, but that is not recommended.

// ✖︎ also not recommended - using a for…in loop to repeat a test 
@Test func includesCryptoNames() async throws {
    let cryptoNames = [
        "BTC",
        "ETH",
        "CryptoA",
        "CryptoB",
    ]

    let data = try await GetData()
    for cryptoName in cryptoNames {
        let currency = try #require(data.first(where: { $0 == cryptoName } ))
        #expect(currency == cryptoName)
    }
}

Let’s try to use the Parameterized test function!
Changing it into a parameterized @Test function would be something like this:

// ⚪︎ recommended
struct CryptoCurrencyTests {
    @Test("Check master contains the correct cryptos", arguments: [
        "BTC",
        "ETH",
        "CryptoA",
        "CryptoB",
    ])

    func includes(cryptoName: String) async throws {
        let data = try await GetData()
        let currency = try #require(data.first(where: { $0 == cryptoName } ))
        #expect(currency == cryptoName)
    }
}

Running Swift Testing via Command Line

Just like XCTest, we could also use Swift Testing in a command line so it could be usable in projects with CI/CD. Please use this command:

swift test

Migrating from XCTest

Actually, we could use Swift Testing alongside with XCTests. When we have similar XCTests, we could consolidate those into a parameterized @Test function. And then finally, remove the test from the names of the test cases.

Conclusion

Personally, I like Swift Testing more than XCTest. Swift Testing has improved a lot of things compared to XCTest, and would make it easier to create unit tests than before. Swift Testing can only be used from Xcode 16, so if you have not updated your project to use Xcode 16 just yet, you might have to wait for a little bit to start using Swift Testing.

That’s all. Thank you so much for staying!
I hope you enjoyed reading this article 🙂

References:

The next article will be by @Yani. Please look forward to it!

From Good to Great: Evolving Your Role as a Quality Consultant

Fri, 13 Dec 2024 15:35:03 GMT

This post is the second article for Day 10 of Mercari Advent Calendar 2024, brought to you by @Udit, an Engineering Manager (QA) at Mercari.

This blog is based on my recent presentation at the inaugural edition of Tokyo Test Fest (TTF) 2024 and is also inspired by quality leaders and speakers from around the world.

Quality Consultant

Quality Consultant, sometimes being an abstract term, allows you to act as a Quality Consultant or architect across the organization, projects, teams, domains, etc.

Quality Consultant, Quality Advocates, Test Architects, or Quality Experts: Different companies follow different nomenclature, but mainly surround similar skill sets.

Few companies have such a designation; for the rest, it is an implicit part of Senior QA roles. The idea of this blog is to give you insight into what works and what doesn’t work when you would like to evolve yourself as a great Quality Consultant.

How did I become a Quality Consultant?

I am not a Quality Consultant by designation, but I play that role in my day-to-day work life.

I started as a developer, then evolved into testing and automation. I learned about different frameworks and programming languages, automation across various layers and platforms, and gained experience in different types of projects.

Quality Consultant Role & Skillset

The Quality Consultant role and focus areas differ from company to company, between product and service industries, and whether you are acting as an external or internal consultant, along with a common set of necessary skills which plays a key role.

Potential Career Paths

Here are some potential career paths and routes that can help you evolve as a Quality Consultant within your organization by enhancing your skills and capabilities, and by taking on a larger role or making a bigger impact beyond your existing position.

The "+" indicates the skills you may need to elevate to move from one position to another.

The "-" indicates less emphasis on those skills (though you still need to be aware of them) and instead focuses on strengthening your existing skills and focus areas.

Test Pyramids

Now, as a Quality Consultant working on any new or existing project, it’s important to evaluate the current state of the test pyramid and try to implement one that is desired or close to desired for long-term effectiveness and advancement.

When Projects Go Wrong

The projects usually go wrong when there is more coverage at UI level but less at Unit level, or when there is decent coverage at both UI and Unit level, but almost no coverage at Service level.

When Projects Go Really Wrong

This is again another and very common example when the projects go really wrong, and should be avoided if you are responsible for Quality for such projects.

The Pyramid & Shift-Left

In the usual test pyramid, moving testing early—i.e., moving down in the pyramid—helps achieve faster, easier, and cost-effective testing.

Agile Test Automation Pyramid

Next is the agile representation of the test pyramid across the UI, Service, and Unit layers, where each layer has its own significance in testing. For example, UI layer represents E2E user journeys or critical user flows, Service layer include testing with both real and mocked data, and Unit layer include unit tests.

Now one of the idea is to break down the middle service layer into API, Contract and Component level of testing.

UI & API testing can cover system integration testing, real use cases close to Production

Contract testing is a software testing methodology that tests the interactions between different microservices or software components based on the contracts between them. In contract testing, each service or component is given a contract, which defines how to work with the service and which responses to accept.

Component testing validates the components in isolation, also known as integration testing.

The above figure represents the ideal types of automated test suites that can be targeted across each layer.

Transitioning to SDET and/or Quality Consultant

Things to remember:

It’s important to focus on skill diversification, learn the implementation of test pyramids, embrace shift-left testing and pipeline integration, and be selective while also developing the soft skills necessary for better communication across the organization.

Issues like silos or QA as after thought, heavy reliance on manual testing, redundant execution of regression tests, and inconsistent frameworks can lead to quality concerns, maintenance and scalability problems.

Thank you for reading! Embark on an exciting journey with us to revolutionize the way we approach quality and become a valued contributor at Mercari!

New Production Readiness Check experience in Mercari

Fri, 13 Dec 2024 11:00:44 GMT

Introduction

This post is for Day 9 of Mercari Advent Calendar 2024, brought to you by @mshibuya, a Tech Lead of the Mercari Marketplace Site Reliability Engineering (SRE) team.

My team Marketplace SRE is part of the Platform Division, which provides the Platform for the Mercari Group as a whole. This article discusses improvements made to the process called Production Readiness Check, which supports the reliability of our services and how it changed the developer experience.

The importance of services having adequate reliability is widely recognized. However, the efforts required for this can be tedious and labor-intensive, leading to a slower development speed due to the existence of this production readiness process. I will describe what aspects of the Production Readiness Check process were improved and what kind of developer experience we aimed to create as a result. I hope this will be useful for those who are undertaking similar initiatives.

About Production Readiness Check

At Mercari, there is a process called Production Readiness Check (PRC). This is a checklist of criteria that newly developed products or microservices must meet, and without passing this they cannot be operationally launched in the production environment.

Besides an introductory blog article, although not the latest, the checklist items themselves are available on GitHub.

Mercari broadly adopts the microservice architecture. In large-scale services such as the Mercari marketplace app and the mobile payment service Merpay, many feature additions are made in the form of newly-developed microservices. New products like "Mercoin" and "Mercari Hallo" also take the form of a microservice on the same infrastructure as "Mercari" and "Merpay." Hence, the launch of new microservices happens frequently. Following the DevOps principle of "You build it, you run it," the individual microservice developer teams are responsible for ensuring reliability in the production operations.

Microservice development teams may not always be familiar with launching new services or ensuring reliability. The purpose of the Production Readiness Check process is for developer teams to autonomously launch microservices while ensuring necessary reliability.

Challenges to Solve

The Production Readiness Check has played an indispensable role in ensuring that services developed at Mercari have sufficient reliability (i.e. production-ready) to operate under real user traffic. However, this process of checking for production readiness comes at a cost to developers’ time.

The Production Readiness Check process at Mercari begins with creating an issue that includes the checklist and ends with the closing of the issue.Over the last 5 years, it’s taken an average of 35.5 days to complete the PRC—although this is a reference value, since actual work does not occur throughout the entire period from issue open to close.

Developer interviews conducted by the Platform Division revealed that there were many complaints about the Production Readiness Check process. Examples include:

Did PRC as well, lots of “copy this, paste this, take a screenshot of this…”
Overall straightforward, just PRC was a pain

PRC, takes about 4 weeks

Takes a lot of time
Personal opinion is that 1-2 sprints could be cut by simplifying the PRC process

Too many things to check, some things are hard to understand how to verify

One of the least desirable tasks. I understand it’s necessary.

At the Mercari Group, speed in launching new products and adding features to existing products is more important than ever. Therefore, speeding up this Production Readiness Check process and reducing the delivery time was an urgent task.

Developer Experience with the Existing Process

Here I will present a typical experience before the improvements in the Production Readiness Check process, using the launch of a new product as an example. This example is fictional, so please consider it as a possible worst-case scenario a developer could have experienced.

Let’s say that the Mercari Group decides to launch a hypothetical new product. This is a high-criticality product integrated with the Mercari marketplace app.

A development team is formed with a goal of launching this new service within six months. The team first clarifies the product requirements and designs the system implementation, compiling it in the form of a Design Doc. Based on the completed design, they proceed with the implementation of the actual application code. They are able to finish implementing almost all the functions by the fifth month, just before the public launch.

While the team prepares for the actual product release, setting up the infrastructure for production use, they realize that they need to go through the Production Readiness Check process. The team, recognizing that meeting these requirements is mandatory for releasing the product, does their best to finish, but due to the sheer number of requirements and aspects that were not included in the initial design, they struggle.

As a result, the team took two months to complete the Production Readiness Check, leading to a delay in the product launch and a lost opportunity to release the product early and gain feedback from users.

Solution

Check Automation

One primary factor contributing to the labor intensity of the process is the sheer number of items to be checked, which is steadily increasing due to learnings from past incidents.
The number of checklist items for typical services has increased from 62 in the publicized version, to 71 in the latest internal version, an increase of nearly 15% over approximately three years.

Moreover, while the items included in the checklist define the desired endstate, they rarely guide teams how to get there, further slowing developers down as they investigate.

To solve this problem, we introduced automated verification of checks in the Production Readiness Check process, including scanning application code and infrastructure configuration. We have automated almost half, about 45%, of the checklist items, and plan on growing this number in the future.

Not only has this made it easier for developers to conduct checks for their service, but these automated checks also make it easier for developers to understand how to fulfill the requirements, facilitating faster and easier mitigation actions.

Enhancement of Existing Platform Components with Production Readiness Check Compliance

As has been presented on past occasions, Platform Engineering is widely practiced at Mercari. Under the concept of enhancing developer productivity through self-service-focused Platforms, the Platform Division has built and provided many components.

During the process of identifying the reasons for the high burden of the Production Readiness Check process, we realized there was a gap between the requirements and the functions of the components actually provided by the Platform.

Mercari’s Platform offers various components throughout all stages of the software development life cycle (SDLC), allowing developers to efficiently achieve their necessary objectives. We identified ways to improve the platform offerings themselves, such as tools for automated Continuous Integration / Continuous Delivery (CI/CD), to fill in the gaps.

Additionally, as a more important and cost-effective improvement, we enhanced documentation to clarify the Production Readiness Check requirements that can be met by these components.

An insight gained through these efforts is the importance of integrating such components to create a comprehensive developer experience, towards the unavoidable Production Readiness Check process when building microservices. We believe that by not only providing components but also improving the check process itself, we have created a situation where a bi-directional feedback loop can function.

"Shift-Left" Approach

In this context, "Shift-Left" is a concept often used in the context of software testing or security, referring to moving activities like test execution to an earlier stage (i.e., "left side" in a timeline diagram).

In the aforementioned new product development example, the team attempted to complete the Production Readiness Check process in a short period just before releasing the product, encountering difficulties due to the high labor intensity. I personally refer to these situations as "the last-minute summer homework problem," but I believe this is due to structural issues more so than the fault of any individual team members. Launching a new product involves various challenges and difficulties, and, while focusing on these, it is inevitable to postpone things known to be important but not immediately needed.

To address this problem, I thought improvements at a systemic level were necessary. Now, with automation achieved, the team can perform the checks for automated items repeatedly to incrementally meet the requirements. Also, by adopting the expanded Production Readiness Check compliance through existing components, they can start fulfilling the requirements in advance without much effort. Then finally, by ensuring the team is aware of these measures from the early development stage, we can prevent work being concentrated in a short period just before release.

However, just informing the existence of such new processes and solutions has its limits. Therefore, by embedding them into another established process that is guaranteed to occur at the start of every development, we ensure that teams in the early development stage can recognize its existence without omission. Mercari’s culture is to create a Design Document for new services to be reviewed by stakeholders. To ensure that Production Readiness is considered earlier in the SDLC, the Design Document template was expanded to include details about these production checks.

As a result of these "Shift-Left" measures, developers can become aware of these requirements from the design stage, long before actual development or infrastructure setup happens, and take meaningful actions toward the Production Readiness Check process earlier.

Developer Experience with the New Process

The following illustrates what sort of experience we want to achieve with the improved Production Readiness Check process, incorporating automation.

Let’s go back to the hypothetical development of a new product, but with the new process in mind.

First, as a result of Shift-Left, the team becomes aware of the Production Readiness Check process at the earliest stage of a six-month development period while designing and creating the Design Doc. Understanding the requirements that need attention earlier allows them to consider options from the design stage, such as discussing with stakeholders about changing product requirements to meet the Production Readiness Check requirements.

By the fifth month, with the product launch coming closer, the team begins preparations for the Production Readiness Check process. Having selected appropriate Platform components to meet requirements, the team minimizes additional changes or efforts required to meet them.

The automated checks significantly reduce the labor to verify and fix compliance with Production Readiness Check items. Consequently, the team completes the Production Readiness Check process within a month, able to deliver value to users early and refine the product through feedback.

Future Plans

As outlined above, the Production Readiness Check process has been improved and is starting to be utilized for checks before actual microservice releases. However, there is still room for improvement of existing components to be more compliant on Production Readiness Check requirements, and automation to increase the applicable cases.
To achieve a higher developer experience, both of these aspects are expected to be areas of focus for the foreseeable future.

What lies ahead as these improvements advance?
Personally, I consider it ideal to eliminate the idea of "conducting checks" altogether. In a world where almost all requirements are inherently met through the functionalities and components provided by the Platform, developers could inherently build and operate reliable services without having to think about it.
I want to consider how we can achieve the ideal Platform where we don’t need to care about such reliability requirements, even though the journey may be a long one.

Conclusion

In this article, I explained the overview of the Production Readiness Check process at Mercari, detailed what improvements were made to the process, and illustrated what kind of developer experience it was possible to create as a result.
Tomorrow’s article will be by sintario_2nd. Please continue to enjoy!

From Embedded to Standalone: A Newcomer’s Transition to Hallo Flutter App Development

Wed, 11 Dec 2024 11:00:45 GMT

Introduction

Hi, my name is Cherry. I’m so excited to be part of this blog series!

Let me introduce myself first. I joined Mercari in October this year and am now a Flutter engineer on the Mercari Hallo mobile team. Before this, I was a native Android app developer and spent about two years migrating a native app to Flutter using the add-to-app embedded app approach. Since the Mercari Hallo app is a standalone Flutter app, I’ve faced significant challenges in transitioning to the different development focus, and the new project architecture that comes with it.

In this blog, I will demonstrate the challenge and highlight a few points about how Mercari Hallo’s onboarding process and documentation helped me overcome. I hope this blog will offer insights for readers considering starting Flutter or migrating native apps to Flutter.

Challenge: The New Development Focus

During onboarding, I noticed two major differences compared to my previous experience.

Simpler project architecture
Deeper focus on Flutter-specific development

Simpler Project Architecture

Overall Architecture

In embedded development, we can either fully replace a page or replace only a specific part of a page with Flutter. However, the latter requires precise control over the native page’s lifecycle via bridges, making it unsuitable as a solution for large-scale apps. Therefore, I will focus on the first approach.

An embedded Flutter project is created as a module, with the host iOS and Android apps that reference this module as a dependency. This setup requires separate maintenance for the Flutter module and the host projects. While a standalone Flutter project eliminates the need for separate management.

Business Logic Complexity

Routing
In an embedded app, handling mixed stacks of Flutter and native pages is an unavoidable challenge. I have encountered two solutions:

Routing managed by the native side
Routing managed by both native and Flutter
This method requires stack information synced via bridges. For typical apps that use a navigation bar the sync could be complex, as each navigation item usually holds an independent stack.

On the other hand, a standalone Flutter app rarely needs to deal with this complexity. Mercari Hallo app uses go_router to manage page routing. It also leverages StatefulShellRoute to build StatefulShellBranch, enabling easy management of the stack for each tab in the bottom navigation bar.
This is a sample routing structure of Mercari Hallo:

routing root
  |__ homeRoute: StatefulShellRoute
        |// branches corresponding to bottom navigation
        |__ timelineBranch: StatefulShellBranch
        |__ favoriteBranch: StatefulShellBranch
               |__ offerRoute: GoRoute
        // switch tabs by statefulNavigationShell.goBranch()

Bridges
In an embedded app, a large amount of bridging is often required to handle data sharing between multiple page engines. But a standalone Flutter app only needs to define custom bridges in a few cases, such as handling deep links or interacting with native interfaces for creating files in the file system.

Deeper Focus on Flutter-Specific Development

In my previous experience working on both Android and Flutter, the focus was on tackling the hybrid architecture. But development for the Mercari Hallo app pays major attention to Flutter and Dart

Encouraging to Use the Best Practices of Dart

Here’s an example I encountered during a code review. In Kotlin, the common approach is to construct a list using MutableList<T>() then update elements and apply transforming or filtering through methods like map, filter, and finally use toList() to gather the results into a new list. This became a habitual way of writing code for me:

return myList
    .map((element) => element.copyWith(property: newValue))
    .toList();

return myList
    .whereType<MyFilteredListType>()
    .toList();

However, Dart provides a collection literal <ListType>[] syntax and allows using the spread operator (…) to insert other lists directly into the content of the list. It also supports embedded for loops or if-else control flows. As a result, I refactored the code to follow Dart’s preferred style:

return <MyListType>[
  for (final element in myList)
    element.copyWith(
      property: newValue,
    ),
];

return <MyFilteredListType>[
  ...myList.whereType<MyFilteredListType>(),
];

Emphasizing UI Testing

After joining Mercari Hallo, more attention was placed on UI (widgets). This is because the code structure minimizes the focus on the data and business logic layers, which I’ll discuss in the next section. With the focus shift, unit testing for widgets became essential. Actually, most of the tests are concentrated on widget tests.

Widget Testing
When business logic is tied to UI states, it needs to be covered within widget tests. The WidgetTester has functions such as tap and drag, which are used to simulate user interactions and trigger different UI states. The displayed data is then used to verify the underlying logic.

Golden Testing
Mercari Hallo app uses golden test to check UI intuitively. The flutter test --update-goldens --tags=golden command generates golden images, and the matchesGoldenFile function checks for differences. These images cover both light and dark modes, as well as large and small screen sizes.

Adopting React-Like Architecture

When doing native Android development, I used the model–view–viewmodel (MVVM) architecture, keeping View, ViewModel, Repository, and Data layers separate. Among Flutter’s state management solutions, BLoC is probably closest to MVVM, as it updates the state through events from the UI and populates backend data to UI as well. This is similar to the ViewModel’s two-way binding.
However, Mercari Hallo adopts a React-like architecture with flutter_hooks:

Components compose a page

Hooks manage state, with heavy use of custom hooks
A typical page architecture in Mercari Hallo might look like this:

./lib/src/screens/
|__ hoge_screen/
     |__ components/  --> The UI components for the page
           |__ hoge_header.dart
           |__ hoge_content.dart
           |__ hoge_footer.dart
           |__ hoge_error.dart
     |__ hooks/  --> The custom hook
           |__ use_hoge_screen_state
     |__ gen/

This structure organizes logic around pages and components, rather than separating it into distinct layers. It also ensures a unidirectional flow of state passing down the Widget tree.

Work through

As for how I’ve been working through the above challenge, the following factors greatly helped.

During Onboarding

Comprehensive README Documentation

Clearly lists every possible step, avoiding omissions, including directory movements, environment variable setup, etc.
Provides separate steps for different shell environments. (e.g. bash, zsh)
Highlights any project-specific, recommended, or non-standard practices compared to official documentation.
Maintains a troubleshooting section.
Encourages team members to update the documentation actively.

Monorepo Flexibility

With the monorepo, engineers can freely set up the environments for other ends based on the documentation, significantly reducing the cost of understanding the entire project.

During Development

Actively Adding Custom Linter Rules:
We not only adopt many Dart linter rules but also have a hallo_linter package for custom linter rules to enforce specific guidelines in certain scenarios.

These extra rules help enforce the use of standardized Dart code across the team.
Actively improving CI/CD processes
Emphasizes best practices in code reviews

Conclusion

Shifting from embedded Flutter development to a standalone project like Mercari Hallo was both challenging and rewarding. It required adapting to new architectures and focusing more on Flutter-specific features.
This experience helped me grow technically and showed the value of good documentation, monorepo flexibility, and clear coding standards. I hope my journey offers helpful insights to others exploring Flutter or migrating native apps. Thanks for reading!

We hope this article has been helpful to your projects and technical explorations. We will continue to share our technical insights and experiences through this series, so stay tuned.

Also, be sure to check out the other articles in the Mercari Advent Calendar 2024.
We look forward to seeing you in the next article!

The React Profiler Demystified

Tue, 10 Dec 2024 12:00:11 GMT

This post is for Day 6 of Mercari Advent Calendar 2024, brought to you by Sam Lee from the Mercari Seller 3 team.

When building web applications, performance can make or break the user experience. With large applications like mercari.jp, we as engineers have to be more mindful of its performance. Whenever there is a stutter or jank, I always find myself not knowing where to start. Was that just a slow API call or was there an expensive calculation? This is where the React Profiler comes in. It’s a tool that can help you pinpoint performance bottlenecks easier than before. In this post, I’m going to explain what the React Profiler is and dive into a hypothetical example.

What is the React Profiler?

The React Profiler is part of React’s Developer Tools browser extension that helps you measure the performance of your React app. When an application becomes complex with many components re-rendering in response to state or prop changes, the Profiler gives you the ability to zoom in on these re-renders. It breaks down why these re-renders are happening and highlights performance issues like excessive renders or unnecessary computations.

A hypothetical example

Telling you the different parts of the profiler probably won’t be too fun, so let’s learn by example and see how the React Profiler can be used.

Let’s assume that you’re working on a hypothetical React application and tinkering with the development build in your free time. This is when you notice a brief but annoying jank after clicking on a button that displays a list of items.

What happens on the development build may not happen on the production build, so you head on over to your production site and open up the Chrome DevTools’s Performance tab. You hit record, click the button in question, and then watch as the timeline loads…only to find that there’s a whopping 100 milliseconds between when you click the button and the next UI update—that is 10 frames per second (FPS) when playing your favorite game.

In order to find out what causes this, you redo the whole thing again but now with your handy React Profiler. Hit record, click the button and hit stop.

You filter out all commits, which are changes that React applied to the DOM (Domain Object Model), that took less than 20 milliseconds, because they’re likely too small to matter.

You only want the “frames” (commits) causing your app to drop to 10 FPS. One particular commit stuck out towering over everything else.

_{A commit bar graph displaying the durations of each commit by height. Commit is the phase when React applies changes directly to the DOM.}

You clicked on the commit which updated the Flame graph, a hierarchical visualization showing the time it took to render a component relative to its children. Invented in 2011 (quite recent!) Flame graphs were created originally for showing the CPU usage of function calls in MySQL.

_{A flame graph with the top bar being the parent and its children below it. Duration of a render is shown by its width and denoted by the right most number on the bar.}

The component highlighted in yellow is what caused the particular commit and the slow render. Upon closer inspection of the time it took to render, you see that the 0.9ms—the time it took to render just the parent component is only a tiny fraction of 87.8ms—the time it took in total to render the parent component and its children. It’s not that the component is inefficient; it’s simply trying to render too many children at once, causing the component to take 87.8 milliseconds!

There are multiple potential solutions. One solution is pagination of the list—displaying the list one manageable page at a time. Another option is to virtualize the list—rendering only a portion of the list at any given time, depending on what’s visible on the screen.

You then pitch the issue, cause, and solutions to the team.

Final thoughts

I hope that example helped in demystifying just a bit of what the React Profiler is. Do keep in mind that performance bottlenecks come in all shapes and sizes. Some are caused by unnecessary re-renders, others by inefficient rendering strategies or sheer scale. In my personal experience, re-renders of not just a component but the entire page are the most common. But of course your mileage may vary and knowing how to approach these problems can make all the difference.

Tomorrow’s article will be by @cherry. Please look forward to it!

Insights from FinOps X Europe 2024: A Scholar’s Journey

Mon, 09 Dec 2024 14:09:02 GMT

Introduction

In this article, I share my experience attending FinOps X Europe, the largest event in the FinOps industry, and how I got the opportunity to participate through a scholarship program. I’ll walk you through the key takeaways from the conference, including the latest trends and developments in FinOps, as well as the invaluable networking opportunities and tool discoveries that enriched my professional journey. Beyond the official sessions, I’ll recount the unique experiences and insights gained from interacting with a global community of FinOps practitioners. I hope this article will pique your interest in FinOps and perhaps inspire you to consider attending a future FinOps X event, where we might have the chance to meet and exchange ideas.

What is FinOps X?

FinOps X is a global conference series organized by the FinOps Foundation. This event features industry-leading talks and offers interactive sessions. It provides a unique networking opportunity for FinOps practitioners to connect and share experiences. Notably, for this European event, we booked the entire hotel for the duration of the conference, creating an immersive and exclusive environment for attendees to fully engage in the FinOps experience. As a FinOps engineer, I often found it challenging to find common ground with other stakeholders in my daily work environment. However, during this event, I felt a sense of comfort and belonging, as if I had returned home. The conference provided a rare opportunity to be surrounded by like-minded professionals who truly understand and appreciate the intricacies of FinOps.

Beyond Borders: Navigating FinOps X Europe with Scholarship Support

Attending overseas conferences can be challenging due to work schedules, travel fatigue, time differences, and high costs. The financial burden is often the most significant obstacle, even when companies cover some expenses.

However, receiving a scholarship changes the equation. I was invited to FinOps X Europe with a scholarship, which not only eased the financial burden but also recognized my contribution to the FinOps community especially to Japan and South Korea.

Scholarship program details have been changed. But it remains an attractive opportunity for people who want to enter this industry. For current information on scholarship opportunities, visit the official FinOps Foundation website.

Conference Highlights: Key Takeaways

The FinOps X Europe conference provided valuable insights into the latest trends and developments in the FinOps field. Here are the key takeaways:

Expanding FinOps Scope:

The first day’s keynote, which is now available on YouTube, presented a paradigm shift in FinOps thinking. It proposed extending FinOps scope beyond the public cloud to private cloud, data center, SaaS and licenses.

FinOps in AI Services:

With the rapid cost escalation associated with AI services, discussions began on how FinOps experts can contribute to managing and optimizing these expenses. This highlights the evolving role of FinOps in emerging technologies.

FOCUS 1.1 Release:

The FinOps Foundation introduced the latest release of their Format, FOCUS 1.1. This update to the FinOps framework was a significant point of interest, likely offering new guidelines and best practices for practitioners.

Emphasis on SaaS Management:

While cloud providers (AWS, GCP, Azure) have been the primary focus of FinOps, there was an intriguing discussion about the need for FinOps to pay more attention to Software as a Service (SaaS) costs. This is particularly relevant as SaaS expenses are growing rapidly.

Japan’s SaaS Market Growth:

While Japan’s SaaS market is smaller than Europe and the United States, it is showing rapid growth. This trend underscores the importance of applying FinOps principles to SaaS management in the Japanese context.

Networking and Community Building: The Hidden Gem of FinOps

The true value of the FinOps X event continued long after the official sessions ended. Each evening, participants from various countries gathered over dinner to share their experiences and knowledge. These gatherings went beyond simple networking, becoming a platform for practical problem-solving.

Attendees openly discussed FinOps-related challenges they faced in their companies and received advice from others. For instance, when one participant expressed difficulties with cloud cost optimization, others shared strategies that had been successfully implemented in their organization. This exchange provided vivid, on-the-ground experiences and insights that are often hard to obtain from formal presentations.

Exploring the FinOps Tool Ecosystem

The FinOps X venue was filled with numerous SaaS companies sponsoring the event and showcasing their solutions. This presented attendees with a valuable opportunity to get a comprehensive view of the latest tools and technologies in the FinOps field.

Before the event, I was familiar with only a few well-known tools. However, through this experience, I realized that there are numerous solutions supporting various areas of FinOps. I had the chance to directly experience tools specialized in cost optimization, resource management, predictive analytics, report generation, and more, while receiving expert explanations.

This experience went beyond simply discovering new tools; it provided concrete ideas that could be applied to FinOps practices. It will be immensely helpful in selecting and implementing appropriate tools that meet our organization’s needs in the future.

Conclusion

FinOps X Europe was a transformative experience that broadened my perspective on FinOps. The power of networking and community building stood out, providing invaluable opportunities for idea exchange and problem-solving.

As a FinOps engineer from Japan, I gained insights that will enhance my practice and contribute to the growing FinOps community in Japan and beyond. However, I couldn’t help but feel a bit disappointed by the limited representation from Asian countries at the event. This underscores the need for greater engagement and participation from the Asian FinOps community in global events.

The conference reinforced that FinOps is about optimizing value and driving innovation, not just cutting costs. Looking ahead, the next FinOps X is scheduled for June 2025 in San Diego, USA. I hope this article has sparked your interest in FinOps, and perhaps we’ll have the chance to meet at a future FinOps X event. It would be especially great to see more attendees from Asia next time.

The Race Condition in multiple DB transactions and the solutions

Mon, 09 Dec 2024 10:00:42 GMT

This post is Merpay & Mercoin Advent Calendar 2024 , brought to you by @timo from the Merpay Balance team.

This article is going to discuss the race condition happening when using multiple database (DB) transactions in one API / request. And give you some insight of how we overcame it.

Background

The Balance team is responsible for storage balance / debt for the Merpay users and the related accounting books.

When a user buys something from Mercari or pays in a store using the Merpay Smart Payments (メルペイのあと払い) option, our service creates records to track user’s debts and needs to repay to Merpay before the deadline.

It’s normal that users repay all the debts at the same time.
We are not limited to the number of debts when users repay, so the request might look like the following:

Request: {
    idempotencyKey: "foo",
    CustomerID: 123,
    repayingDebts: {
        {amount: 100, ID: "AAA"},
        {amount: 200, ID: "BBB"},
        {amount: 300, ID: "CCC"},
        ….
        // the number of repayingDebts is not limited
       },
}

It’s common to use a DB transaction to ensure consistency when executing write operations. However, database engines usually have a maximum size of insert/update in one transaction.
Cloud Spanner, which is used in Merpay as the database service, has limitation with Mutations per commit (it was only 20,000 Mutations allowed as of 2021/05). Since there are many records to be inserted/updated for one debt, it was very easy to hit the limitation and get the error.

Multiple DB transactions in one request

To work around the mutation limitation, we tried to break down a single DB transaction into multiple ones. For example:

1st transaction : Insert the received repayingDebts into tables
2nd transaction: Execute the repaying
3rd transaction: Mark the status as repaid

Note that for the “4.Create & update associated records” in the 2nd transaction, it creates an independent DB transaction in each loop to handle all of the debts and executed in parallel. Without doing that, the performance would not fulfill our service level objective (SLO).

One problem fixed, but another came – Race condition

Everything looked good at the beginning, but we got another new trouble later that our system detected the inconsistency of our data. It happens when two requests (A and B) trying to repay the same debts.

In the following example, request A and request B do the parallelProcess at the almost same time, request A finishes the 1 ~ 3 set, request B finishes the 4 ~ 6 set. In this case, request A can not repay the 4 ~ 6 set anymore because the amount of debt has been repaid by request B, so it returns INVALID_AMOUNT. Request B has the same situation, in the end it leads to deadlock.

The race condition happened once or twice a month, it triggered the inconsistency alert and our on-caller needed to recover it manually. The related records might be updated anytime and it makes the fixing query operations become more complicated. It took about half a day for the manual operation, which affected our team performance.

Possible Solutions

To solve the race condition, we considered some solutions as following:

Rollback mechanism

When the race condition happens and is detected, roll back the status and amount to the values before repaying. It can be imaged as a manual operation but executed by programming.

Lock mechanism

Since the race condition occurs when two requests repay the same debts, it can be guarded by only allowing one request to process repaying, blocking others until the request finishes.

Merge into 1 DB transaction

Going back to the one DB transaction can also prevent the race condition from happening.The root cause is the limitation of mutations. Therefore , a method is to find the most mutation usage and do it asynchronously, to make the total number of mutations under the limitation.

We evaluated the pros and cons for each solution. By considering our database schema design and business requirements, we chose the Lock mechanism.

Challenges of lock mechanism

Challenge 1: Design the key of the lock

How to decide the key of the lock depends on what you want to protect.
In our case, our target is: only one repaying request for the same debts can be processed at the same time for the same customer. Other requests with different idempotency keys will be rejected.

So the schema to store lock information is designed as below:

CREATE TABLE Locks (
  HashKey STRING(100) NOT NULL,
  CustomerId INT64 NOT NULL,
  IsLocked BOOL NOT NULL,
  IdempotencyKey STRING(100) NOT NULL,
  CreatedAt TIMESTAMP NOT NULL,
  UpdatedAt TIMESTAMP NOT NULL,
) PRIMARY KEY(HashKey, CustomerId);

Note that using HashKey (same debt IDs generate the same HashKey) and CustomerId as PRIMARY KEY to ensure that only one request can use the lock at the same time.

Challenge 2: When and how should the lock be unlocked

All of the use cases should be considered because it’s dangerous if any of the records are locked or unlocked unexpectedly.

For example, a request is failed when repaying is an edge case.

Should it be unlocked or not?
=> If all the target records have been repaid, it can be unlocked. Otherwise it can not be unlocked.

(List the use cases and check if working properly)

Challenge 3: Does it need to record all the locking operations?

The lock key is created by the customerID and all repaying debt IDs, and there is a use case that a debt can be repaid partially with a different Idempotency Key. That means: the column IdempotencyKey can be overwritten. We have considered storing all the operations in the database for debug and investigation. However, we found that it’s not really helpful for investigation and it’s enough to output the minimum information to log service for debugging.

Other perspectives

Keep in mind that the responsibility of your service

We also considered the passing parameters from our clients during the design, tried to judge the specific repayment scenario which caused the race condition, and handled it with exception handling. However, it will make your services more complicated and harder to maintain. In our case, we just need to ensure that the debts given from clients exist, and repay it if the amount is sufficient.

The performance in parallelProcess

As the above mentioned, the parallelProcess loops all the debts to repaying. The more records we get, the slower a request goes. Our next goal is to identify how to break through the limit.

Summary

Race condition is a common issue when making processes run in parallel. It is easy to be introduced but painful to be lifted.

We have released our solution for half a year and everything is good and safe for now.
It took one year to solve this problem from design and discussion to implementation. But now our team is not bothered by the race condition issue anymore, and it really saves our time from dealing with the recovery operations.

This article shared our experience of the race condition and gave some possible solutions to you. I hope one of the solutions may inspire you something new! 🙂

Next article will be by @siyuan. Look forward to it!

Streamlining Security Incident Response with Automation and Large Language Models

Sun, 08 Dec 2024 11:00:27 GMT

Background

Effective security incident response is a crucial aspect of any organization’s cybersecurity strategy. The security incident response lifecycle provides a structured approach for handling security incidents methodically and efficiently. By following this approach, organizations can minimize the impact of incidents, recover operations swiftly, and implement measures to prevent future occurrences.

The incident response lifecycle typically compromises the following phases:

Preparation: Establishing policies, procedures, tools, and communication strategies to ensure readiness for potential security incidents.
Detection & classification: Identifying potential security events through monitoring systems and classifying them based on severity and impact.
Triaging: Assessing the incident’s scope, gathering additional information, and analyzing data to understand the incident’s nature and implications.
Remediation & response: Implementing actions to contain and mitigate the security incident, eradicate threats, and prevent further damage.
Recovery, reporting, & learning: Restoring affected systems and services, documenting the incident and actions taken, and learning from the experience to improve future responses through a retrospective analysis.

Understanding each phase enables incident responders to act promptly and effectively. By integrating automation and leveraging Large Language Models (LLMs), the Threat Detection and Response (TDR) team at Mercari enhanced these phases, reducing manual effort and increasing the speed and accuracy of our responses. In this article, we will explain what and how we have achieved these improvements.

Key security incident handling tasks ideal for automation

Manual processes in security incident handling can be time-consuming and prone to errors. To address these challenges, the TDR team developed a security incident response Slackbot that automates repetitive tasks and leverages Large Language Models (LLMs) for tasks requiring contextual analysis (as shown in Figure 1). This automation not only reduces the time spent on routine activities but also enhances the accuracy and consistency of security incidents documentation. In this blog post, we explore the functionalities of our Slackbot, the integration of LLMs, and the significant time savings achieved—between 160 and 250 minutes for a small security incident.

In the rapidly evolving digital landscape, organizations are encountering a growing frequency of security incidents. As a consequence, incident responders are tasked with swiftly setting up investigation environments, coordinating with team members, and meticulously documenting every step of the process. These tasks, while essential, often involve repetitive actions and consume valuable time and resources.

When a security incident occurs, the incident responder has to set up a proper environment to start handling the incident, for example:

Establish Communication Channels: Set up a dedicated platform for real-time collaboration.
Create Documentation Structures: Organize folders and documents to store investigation results.
Assign Tasks: Delegate responsibilities and track progress through task management systems.
Manage Access Rights: Ensure all relevant team members have the necessary permissions.

Throughout the security incident handling process, additional team members may join, requiring further administrative actions. Moreover, documenting investigation results, root causes, impacts, and countermeasures demands careful attention to detail. These manual processes are not only time-consuming but also susceptible to human error. To enhance efficiency and accuracy, TDR developed a security incident response Slackbot that automates many of these tasks. By incorporating LLMs, TDR also automated tasks that traditionally require human analysis.

Figure 1. Security Incident Response Automation.

Automating Security Incident Response Tasks

Our security incident response Slackbot automates several key tasks across different stages of security incident management. In Table 1, we detail these tasks and the time savings achieved.

Security incident creation
Task	Steps	Time
Create folders to store the incident report and artifacts.	Locate the correct folder structure. Create new folders.	3-5min
Create a document for the incident report.	Find the correct template for the incident report. Copy the template to the correct folder. Update the document with the initial incident specific details.	5-10min
Create tasks in Jira for the incident.	Find the correct project. Create the initial tasks.	5-10min
Create a private channel in slack and pin the relevant documents.	Navigate to slack. Create a new channel. Pin the relevant documents as the incident report and the Jira issue.	3-5min
Add relevant members to the channel from a previous initial discussion thread.	Find the correct team members. Add them to the channel.	2-3min
Security incident investigation
Give access to the folders and documents to members joining the slack channel.	Monitor the slack channel for new members. Manually give access to relevant resources.	1-3min per person
Document relevant slack messages in the incident report.	Navigate to the relevant slack conversation to find the message. Copy and paste the message to the incident report. Copy and paste the message link to the incident report. Format the message properly.	3-5min per message
Security incident Postmortem
Create a post-mortem retrospective document.	Find the correct template for the post-mortem retrospective document. Copy the template to the correct folder. Update the document with the incident specific details.	5-10min
Total Time		27-51min

Table 1. Security incidents tasks and time saved by automation and LLMs implementation.

By summing the time saved across tasks, we can observe substantial efficiency gains:

Per incident: Up to 50 minutes saved only for repetitive tasks. Allowing responders to focus on critical decision-making and response activities.
Cumulative: Over time, these savings significantly enhance team productivity and security incident handling capabilities.

Leveraging Large Language Models (LLMs)

Automation significantly reduces the time spent on repetitive tasks. However, certain tasks require contextual understanding and analysis, requiring human intervention. By integrating LLMs into our Slackbot, TDR automated these complex tasks, further enhancing efficiency.

LLMs are AI models trained with a big amount of data. They can understand context, interpret nuances in languages, and generate coherent and relevant text responses. By leveraging LLMs, our Slackbot can perform tasks such as summarizing lengthy discussions, translating languages, and generating detailed reports which require a big amount of time from incident responders.

Challenges

Understand the security incident context.
Accuracy and reliability of outputs.
Handling bilingual communication.
Integration with existing systems.
Computation resource requirements.

Security incident declaration

Before declaring a security incident, responders need to analyze the initial information, understand the context, and determine the appropriate course of action. Crafting a clear and concise description and title for the incident is crucial for effective communication. Finally, determining security incident type, category, severity, and affected assets requires careful consideration.

To address this challenge, TDR leveraged LLMs to:

Do contextual analysis: The LLM processes initial messages and data related to the potential security incident, extracting key information and understanding the situation’s nuances.
Automate description generation: Based on its analysis, the LLM generates a detailed incident description and a descriptive title that accurately reflect the situation.
Assist with security incident classification: It suggests a security incident type and category by comparing the incident characteristics with known patterns and categories.
Estimate impact and severity: The LLM assesses potential impact and severity levels, aiding responders in prioritizing the security incident.
Identify affected assets: It identifies and lists the affected systems or assets by cross-referencing mentioned resources with the organization asset inventories.

Manually could take between 5 to 10 minutes based on the following steps:

Read the initial information of the security incident.
Analyze the context of the security incident.
Write a description of the incident.
Write a descriptive title.
Set a security incident type.
Set a security incident category.
Set an initial impact.
Set an initial severity.
Identify the affected assets.

Security incident reporting and status updates (Daily, weekly, monthly report)

Collecting and organizing information about a security incident, or incidents that occurred over a period is a task which requires large amounts of time. It involves ensuring each incident is summarized uniformly, highlighting key details. Also, responders have to make sure to clearly document actions taken, impact changes, countermeasures, and recommendations that will be later part of a daily, weekly, or monthly report.

To address this challenge, TDR leveraged LLMs to:

Automate security incident collection: The Slackbot gathers incident data from our database for the specified period of time to be sent to the LLM.
Standardize summaries: LLM creates concise summaries for each incident ensuring consistency in format and content.
Generate insights: LLM identifies common patterns, frequently affected assets, and recurring issues.
Generate actionable recommendations: LLM suggests countermeasures and preventive actions based on the analysis. All of them are useful during post-incident activities like retrospectives.

Manually, this could take between 60 and 90 minutes based on the following steps:

Collect security incidents for a given period of time.
Analyze each incident:
- Specify a summary for each incident.
- Specify the impact for each security incident.
- Specify taken actions.
- Specify countermeasures to prevent the incident from happening again.
- Specify recommendations.

Slack channel and Thread Summarization

Reviewing the security incident progression is a task which requires following many threads in a Slack channel every time it is required. Or even to bring new members a quick onboarding. Therefore, it is important to have a tool to provide an overview without overwhelming details.

Challenges addressed were mainly:

Volume of communication: High volume of messages can make it difficult to extract key points.
Contextual continuity: Maintaining the storyline of the security incident as it unfolded.
Identifying critical decisions and actions: Highlighting pivotal moments in the response.

To address this challenge, TDR leveraged LLMs to:

Conversation summarization: The LLM scans through Slack channels and threads, summarizing discussions chronologically.
Key point extraction: LLM identifies significant messages, decisions, and action items.
Contextual linking: The summary maintains the flow of events, showing how one action led to another.

Slack channels and threads summarization and key discussions in chronological order. This function is useful for different purposes, such as:

Security incident retrospective.
Executive summary.
Catching up with the security incident.

Depending on the phase of the security incident and the amount of threads and messages it might be hard for the incident commander to keep track of them. As the saved time is computed based on the amount of information to analyze it is hard to compute a specific number but the average for a small security incident is between 5 and 10 minutes. However, it could be over 1 hour when involved people and tasks increase.

Language interpretation

When working in bilingual environments, teams could face some delays due to language differences. So, ensuring that translated messages maintain the original intent and nuance is important for the previous functions.

Doing a manual translation in the described functions could take between 60 and 90 minutes in total for an analyst who does not know the language based on the following steps:

Identify Japanese messages.
Translate Japanese messages to English based on the context.
Format the messages properly based on the flow of the events.

Integrating Large Language Models into our security incident response processes has revolutionized the way TDR handles tasks that traditionally require significant human effort and time. Through the use of LLMs, TDR saved between 130 minutes and 200 minutes for a small security incident.

Conclusion

The use of Large Language Models frees up human incident responders to focus on strategic decisions rather than administrative tasks. Also, it provides rapid analyses and outputs accelerating the security incident response process. This is a great benefit when handling large volumes of data and communication as they add some delays during the process.
Our incident response Slackbot demonstrates the significant benefits of automating routine tasks and integrating LLMs for tasks requiring analysis. By reducing manual effort, TDR enables security incident responders to focus on critical thinking and decision-making, improving both efficiency and effectiveness.

However, the potential applications of LLMs in security incident response extend beyond our current implementation. As TDR continues to refine our Slackbot, we plan to:

Enhance LLM capabilities: Explore more advanced models for deeper analysis and better accuracy.
Implement Agent-based Incident Response Roles: Implement agents with security incident response roles as incident commanders, handlers, and analysts to support security incident response notifications.
Automate task tracking: Leverage LLMs to monitor threads where high impact tasks are happening to support and keep the incident commander up to date.
Introduce real-time collaboration: Allow LLMs to participate in discussions by providing suggestions or alerts during live incident handling.

Acceptance criteria: QA’s quality boost

Sat, 07 Dec 2024 08:00:42 GMT

Hello everyone! I’m @____rina____, a QA engineer at Mercari.

Welcome to article #1 in the series Behind the Development of Mercari Hallo: Flutter and Surrounding Technologies and day 3 of the Mercari Advent Calendar 2024!

Recently, on November 15, I gave a talk at Tokyo Test Fest, titled "Acceptance Criteria: QA’s Quality Boost." In this session, I talked about how important acceptance criteria is. QA writes this, and it helps in the whole development process, not just in Flutter. It’s also important for the whole team to review it together.

Acceptance criteria is important for teamwork in the development team. If we define it well and share it with everyone, we can really improve quality. In my talk, I also used real examples from our project to show how this process works.

I previously wrote an article about acceptance criteria; it’s only in Japanese, but if you’re interested in reading more, you can check it out here.

In this post, I’ll share a transcript of my talk.

Acceptance criteria: QA’s quality boost

Hello everyone at Tokyo Test Fest! I’m Rina. Thanks for coming today! Let’s get started with my presentation on the topic, "Acceptance criteria: QA’s quality boost."

Our QA Team’s initiative

Now, I would like to talk about one initiative that our QA Team is undertaking. More specifically, on how we document test cases in the acceptance criteria and have the entire development team review them together.

Acceptance criteria are often used in Scrum and Agile development. Before introducing the acceptance criteria, user stories and test cases were separated. This caused misunderstandings, especially during testing.

This new process helps the whole development team! Product managers, designers, engineers, and QAs—we all work better together. Let’s look at the benefits this process has for each team member.

For example, product managers find it easier to check specifications and continue development without missing anything. Previously, issues with the specifications were sometimes only noticed during the testing phase. This activity helps us find and fix mistakes early, so we don’t have to go back and redo our work.

Frontend and backend engineers are able to agree on the implementation plan beforehand, which makes the development go smoothly.

Additionally, by confirming specific wording and display methods on the spot, we can incorporate real-time feedback from designers, leading to higher-quality product development.

For QA engineers, sharing the ways to create test data and executing tests helps to improve our work during the testing phase. Previously, they had to consult developers about creating test data during test preparations. This new approach made it easier to talk about the order of development based on how easy it is to execute tests.

The whole team now understands complex projects better. This makes communication easier. When we have many projects at the same time, it’s easier to see how each project is going. This helps us work smoothly.
Now, let’s take a closer look at how we implement this initiative.

Three simple steps

We did three things.
First, we started including test cases in the acceptance criteria. Second, we changed how we do reviews. Before, we reviewed separately. But now, we review together. Finally, we made it a rule that the whole team participates in the reviews.
Where do you usually keep your test cases? Who uses them and how?

For example, maybe you use a test management tool. Or maybe you use a Google Spreadsheet or Excel file. Test cases are kept in many different places, and people use them in different ways. By sharing test cases with everyone, like with developers and product managers, they become more useful. They are helpful, and when everyone uses them, it’s even better. QA engineers know.

Previously, the connection between user stories and test cases was weak, increasing the risk of missing important tests and leading to having to redo our work later in the development process. Test cases were like a hidden treasure map. Despite being available to everyone, their value wasn’t used. Teams had trouble with their user stories (islands), not realizing the help they needed was right there.

Before, finding the right test was hard. It was like a treasure map with lots of islands. Each island had treasure, but it took a long time to see what was on each one. Now, we have a sign for each island! The sign is the acceptance criteria. It tells us exactly which tests we need for each user story. For example, the sign tells us what the product should do, what it should not do, and how to test it.

This makes it easier for everyone to understand and build a good quality product.

Example: Acceptance criteria

This slide shows an example of our acceptance criteria. We clearly define the test target, the condition for the test, and the expected result.

For example, in the first row, the test target is the display of the title and label. We expect both the title and the label to display "Login". In the second and third rows, we define the expected behavior of the screen based on the condition of the feature flag. Finally, we test it on iOS and Android to make sure it works the same way on both.

With clear signposts (acceptance criteria), our team navigates development more effectively. We’ve seen improved collaboration, fewer errors, less rework, and faster delivery of high-quality products. Everyone understands the goals and how to reach them—together.

Three simple steps to effective reviews

Now, I’ll talk about how we do reviews. Reviews are super important for our team. They help us all agree on what "good quality" means. This way, we can build a really good product. We review the acceptance criteria and test cases together. This helps us avoid mistakes and problems later. It also makes our work faster.

Let’s talk about how we do reviews. It’s really simple! There are three steps. First, we read it out loud. One person reads each acceptance criteria out loud. By doing this, you can see the acceptance criteria and test cases for each user story. The reader explains each item briefly.

Second, we ask questions. After reading, everyone can ask questions. Developers, QAs, product managers, designers—everyone! It’s good to have different viewpoints. For example, "What data will we use for this test?" or "Do we need this part?" Or even, "Will users understand this?" This approach helped us to have a good discussion as a team.

Third, check out. We review and confirm that everyone understands the acceptance criteria.
Then, the review is finished. These reviews help the team agree about quality from the beginning. That’s it! Three simple steps. Anyone can do it!

Let me explain why our new process is effective. We have a specification review to examine the initial requirements. But sometimes, engineers and QAs don’t understand all the details yet. It’s like looking at a picture that’s not clear.

After this, developers write design documents, and QAs write acceptance criteria and tests. In this way, everyone understands the features and user stories much better. Our new review happens after this. Everyone comes to the review with a clearer picture. Like a high-resolution picture! This makes our discussions better and more focused. We can find problems and improve the details together. Having the review later, when we all understand the details, helps us avoid rework. It improves quality and saves time!

But does this review process work for everyone, even without special skills?

We tried this review with team members of all skill levels. For example, I am a QA engineer, and I started doing this in my scrum team. At first, I did it by myself. But my whole team was able to see good results from it. Now, other QA engineers are doing it too. At first, some people were unsure. But now, everyone does the reviews smoothly. So, why does it work for everyone?

The key is understanding together. We write test cases with the acceptance criteria. This way, the whole team sees the same information. The whole team can discuss this at the same level. That’s why it works for all skill levels.
Of course, we can still improve the process. We want to make it even better! However, I believe this review method has great potential. It helps the whole team focus on quality and improves our development process.

So, here are the key takeaways. I talked about using acceptance criteria and test cases to improve quality. By putting test cases with acceptance criteria, everyone can easily see what to test. We write the test cases directly into the acceptance criteria. This helps the whole team build a better product. So, It’s easy to see and connect it to the user story.
While the whole team reviews together, we have good discussions. Everyone understands the product better. These changes help us agree on quality from the beginning. They help us avoid rework and save time. We want to make this process even better! Please give it a try in your teams and see if it improves your quality too.

Thank you for listening.

Keeping User Journey SLOs Up-to-Date with E2E Testing in a Microservices Architecture

Fri, 06 Dec 2024 11:00:19 GMT

This post is for Day 3 of Mercari Advent Calendar 2024, brought to you by @yakenji from the Mercari Site Reliability Engineering (SRE) team.

At Mercari, our SRE team is dedicated to maintaining and enhancing the reliability of our core product, the Mercari marketplace app, by measuring its availability and latency. We establish Service Level Objectives (SLOs) for these metrics and monitor their adherence, as well as whether availability and latency are degrading due to temporary outages or other issues.

To achieve this, our SLOs are based on Critical User Journeys (CUJs). We recently revamped these SLOs, redefining them as "User Journey SLOs" to achieve the following:

Clarify the definition of CUJs.
Establish a one-to-one relationship between each CUJ and its corresponding Service Level Indicator (SLI).
Automate the maintenance of CUJs and SLOs.
Visualize the behavior of each CUJ during incidents through dashboards.

This initiative resulted in a 99% reduction in SLO maintenance time and enabled near-zero time triage, meaning we can now start assessing impact within seconds of incident detection.

This article details the rationale behind revising our CUJ-based SLOs and explains each of the four objectives mentioned above, focusing on how we achieved continuous updates using end-to-end (E2E) tests and leveraged them effectively.

Current Challenges

Before delving into the main topic, let’s examine the two types of SLOs used at Mercari and the challenges they presented. This section explains the motivation and goals behind the User Journey SLO initiative.

Microservice SLOs and Their Challenges

At Mercari, our backend architecture utilizes microservices. For example, user data is handled by the User service, and item data by the Item service. Each domain has its own independent microservice (these are simplified examples and may not reflect the actual implementation). Each service is managed by a dedicated team responsible for its development and operation. Each team sets SLOs for their services and is responsible for meeting these objectives. These SLOs also drive monitoring and alerting, enabling development teams to respond to service incidents.

While defining SLOs for individual services is crucial for teams operating and developing independently, relying solely on these microservice SLOs presents challenges. One of the major challenges is the difficulty of evaluating the product’s overall reliability from the user’s perspective.

Microservices handle specific domain functions. For simple scenarios confined to a single domain, like "editing user information," only one service (e.g., the User service) might be involved. In these cases, assessing SLO’s attainment is straightforward. However, more complex scenarios like "shipping a purchased item" involve multiple services, making it difficult to evaluate the overall reliability of the user journey.

Furthermore, not all APIs within each service are used in each scenario. Development teams may not have a complete understanding of which APIs are used where, as APIs are generally designed for flexibility and reusability. Conversely, frontend developers typically aren’t overly concerned with which service is being accessed.

For these reasons, assessing end-user experience, such as successfully shipping purchased items, becomes difficult using only microservice-specific SLOs. Even if services A, B, and C individually meet their availability targets, the user-perceived availability might be lower. During incident response, an alert from Service A doesn’t necessarily indicate the user impact, hindering prioritization and mitigation efforts.

SRE and SLOs

To address the challenges posed by microservice SLOs, our SRE team monitors our overall marketplace service based on Critical User Journeys (CUJs), independently of the microservice-specific SLOs. CUJs represent the most critical sequences of actions frequently performed by users. However, this approach also presented challenges:

Unclear Definition: The definition of CUJs and the rationale for selecting associated APIs were undocumented, making it difficult to add or maintain CUJs.
Multiple SLOs per CUJ: Directly monitoring the SLOs of each related API resulted in multiple SLOs for a single CUJ, hindering accurate assessment of user-perceived reliability.
Cumbersome Updates: Frequent functional developments and API changes led to high maintenance costs and difficulty in keeping CUJ definitions and their corresponding SLOs up-to-date.
Opaque Impact of SLO Degradation: When SLOs were not met, the impact on users was unclear, making it difficult to prioritize responses and hindering broader utilization of CUJ-based SLOs across Mercari.

Challenge 3, in particular, resulted in a lack of comprehensive maintenance since the initial implementation around 2021, potentially leading to gaps in monitored APIs. To address these issues and enable effective use of CUJ-based SLOs across Mercari for reliability improvements and incident response, we decided on a complete rebuild.

Overview of the User Journey SLO

To address the first two challenges—unclear CUJ definitions and multiple SLOs per CUJ—I’ll explain how we defined and managed CUJs within our User Journey SLO framework and how we established corresponding Service Level Indicators (SLIs).

Defining Critical User Journeys (CUJs)

For User Journey SLOs, we maintained a similar level of granularity to our previously defined CUJs, encompassing tasks like product listing, purchasing, and searching. We revisited and redefined approximately 40 CUJs, covering both major and minor user flows. To address the unclear definition challenge, we documented each CUJ using screen operation transition diagrams, explicitly outlining the expected screen transitions resulting from user actions. We also defined the available states for each screen. A CUJ is considered available if these states are met and unavailable if not. Generally, if the core functions of a CUJ are available, the CUJ is considered available. Secondary features, such as suggestions, that don’t impact core functionality are not considered in the availability calculation.

Defining the SLI

To address the multiple SLOs per CUJ challenge, we defined SLIs to establish a one-to-one relationship between each CUJ and its availability and latency metrics. These SLIs are measurable using our existing observability tools. At Mercari, a single customer operation typically involves multiple API calls, as we generally don’t utilize a Backend for Frontend (BFF) architecture.

Ideally, we would directly measure the success of each screen transition within a CUJ. However, we currently lack the infrastructure for such granular measurement. While we considered implementing new mechanisms, the engineering cost of covering approximately 40 CUJs across all clients (iOS, Android, and web) was prohibitive. We also explored leveraging Real-Time User Monitoring (RUM) data from our Application Performance Management (APM) tools, but sampling rates, cost, and feasibility concerns made this approach impractical.

Therefore, we opted to associate the critical APIs called during a CUJ with the CUJ’s SLI. We categorized API calls within a CUJ into two types: (1) those whose failure directly results in CUJ unavailability, and (2) those whose failure does not. To create more accurate and robust SLIs, we focused solely on those in the first category—the critical APIs—for our SLI calculations.

Using metrics from these critical APIs, we uniquely defined the availability and latency SLIs for each CUJ as follows:

Availability: The CUJ’s success rate is the product of the success rates of its critical APIs. For example, if critical APIs A and B have success rates S_A and S_B, respectively, the CUJ success rate S_CUJ is calculated as :
S_CUJ = S_A　× S_B
Latency:The CUJ’s achievement rate for its latency target is the lowest target achievement rate among its critical APIs. For example, if critical APIs A and B have achievement rates A_A and A_B for their respective latency targets, the CUJ achievement rate A_CUJ is calculated as:
A_CUJ = min(A_A, A_B)

Identifying Critical APIs

To implement the SLI calculations described above, we needed to identify the critical APIs for each CUJ. We considered various methods, including static code analysis, but ultimately chose a hands-on approach using a real application to balance practicality, feasibility, and accuracy. This process involved the following steps:

Proxy and Record: We placed a proxy between a development build of our iOS app and a development environment. We then executed each CUJ, recording all API calls made during the process.
Fault Injection and Validation: Using the proxy, we injected faults by forcing specific APIs to return 500 errors. We then re-executed the CUJ to determine whether the failure of each API resulted in the CUJ becoming unavailable according to our defined criteria.

We used a development build of our iOS app for this process, as it is our most frequently used client.

Communication between our client apps and servers is typically encrypted. Therefore, we selected a proxy capable of inspecting and modifying encrypted traffic. We chose the open-source tool mitmproxy for its interactive web interface and extensibility through add-on development.

The User Journey SLO framework, established with the approach described above, enables us to detect incidents affecting specific CUJs, allowing for immediate identification of the impact scope and faster prioritization of incident response efforts.

Continuous Update and Visualization Using E2E Test

Next, to address the third challenge—cumbersome updates—I’ll explain how we maintain critical API information using iOS end-to-end (E2E) tests. I’ll also describe our dashboard visualization approach, which resolves the fourth challenge—opaque impact of SLO degradation.

The Need for Automation

The Mercari client app undergoes multiple releases each month. Additionally, trunk-based development and feature flags allow us to release new features without requiring app store updates. Tracking all these changes manually is impractical for the SRE team. Manually investigating frequent changes to critical APIs is also infeasible. Undetected changes could lead to monitoring gaps or unnecessary monitoring of deprecated APIs. Therefore, automating the update process for critical APIs is essential to keep up with the changes in application

Automating with iOS E2E Tests

We leveraged our existing iOS app E2E test suite, built using the XCTest framework, to automate the extraction of critical APIs.

Specifically, we implemented each CUJ as an XCTest test case, executable on simulators. Each test case includes assertions to verify the availability of the CUJ according to our defined criteria. This setup automatically distinguishes between available and unavailable CUJs. Furthermore, the test cases are version-controlled alongside the app’s source code.

We developed a mitmproxy add-on to retrieve the list of APIs called during each test and to inject failures into specific APIs. This add-on provides an API to control the proxy, allowing us to manage it directly from our test cases and scripts.

We automated the critical API identification process by scripting the execution of these XCTest tests and controlling the proxy through the add-on. The results, including whether each called API is critical to the CUJ, are logged to BigQuery. Screenshots of the app’s behavior during fault injection are stored in Google Cloud Storage (GCS).

Test results logged in BigQuery are identified by unique IDs, allowing for efficient comparison with previous test runs. We also use Terraform modules, specifically designed for User Journey SLOs, to define and manage SLOs, monitors, and dashboards in our APM system. This allows us to seamlessly integrate changes and easily add new CUJs.

This automation provides several key benefits:

Reduced Maintenance: The process is almost entirely automated, aside from code maintenance for the tests themselves.
Version Control: Both the test cases and the app code are version-controlled in the same repository, ensuring consistency.
Efficient Integration: ID-based management of test results facilitates seamless integration with our APM system.

Ultimately, we created approximately 60 test cases covering around 40 CUJs. This automation drastically reduced the manual effort required, achieving a 99% reduction in maintenance time compared to manual SLO management.

Dashboard Visualization

A key goal of the User Journey SLO framework is to empower teams beyond SRE, such as incident response and customer support, with actionable insights. To achieve this, we needed to present up-to-date information about critical APIs and CUJ behavior during outages in an easily accessible format. We used Looker Studio to visualize this data, providing dashboards that display the list of API calls for each CUJ and screenshots of the app’s behavior during API failures.

Current Status and Future Directions

Through the initiatives described above, we successfully implemented the following for our User Journey SLOs:

Clarifying the definition of CUJs
Establishing a one-to-one relationship between each CUJ and its corresponding Service Level Indicator (SLI)
Automating the maintenance of CUJs and SLOs
Visualizing the behavior of each CUJ during incidents through dashboards

We currently operate SLOs for approximately 40 CUJs, utilizing around 60 test cases. While currently undergoing trial usage within the SRE team, even at this stage, the new SLOs have significantly improved:

Incident detection speed and accuracy
Accuracy of impact assessment
Speed of root cause identification
Overall quality visibility

Quantitatively, we’ve observed the following improvements:

Immediate impact assessment: Achieved near-zero time triage, meaning we can now start assessing impact within seconds of an incident being detected
Reduced maintenance overhead: Achieved a 99% reduction in SLO maintenance time.

Building on these positive results, we plan to expand the use of User Journey SLOs beyond the SRE team, focusing on:

Integrating SLOs into our internal incident management criteria
Leveraging User Journey SLOs to improve customer support responses

Conclusion

This article explored how Mercari implements and operates User Journey SLOs based on CUJs, detailing the specifics of our SLI/SLO definitions and our automated maintenance process using iOS end-to-end testing. We hope this provides valuable insights into managing SLIs and SLOs for complex systems.

Tomorrow’s article will be by ….rina…. . Look forward to it!

Mercari Advent Calendar 2024 is coming up!

Thu, 28 Nov 2024 10:00:40 GMT

Hello! I’m ohito of the Mercari Engineering Office.

We have our annual Advent Calendar blogathon event in December every year and we’ll be hosting it again this year!

We have both Mercari and Merpay/Mercoin Advent Calendar at the same time, so please check out Merpay/Mercoin side as well.

▶Merpay & Mercoin Advent Calendar 2024

What is the Advent Calendar?

The original meaning of Advent Calendar is "a calendar that counts down to Christmas".

We’ll be sharing our knowledge of the technologies used by our engineers at Mercari group. We hope this Advent Calendar will help you to enjoy the days leading up to Christmas.

Advent Calendars 2023

Publishing schedule

This is a collection of links to each article. I recommend bookmarking this page for the prompt update, and it will be very useful if you want to check it out at a later date.

Theme / Title	Author
Google CloudからGitHub PATと秘密鍵をなくす – Token ServerのGoogle Cloudへの拡張	@Security Engineering
Keeping User Journey SLOs Up-to-Date with E2E Testing in a Microservices Architecture	@yakenji
Acceptance criteria: QA’s quality boost	@….rina….
Streamlining Incident Response with Automation and Large Language Models	@florencio
Insights from FinOps X Europe 2024: A Scholar’s Journey?	@pakuchi
React Profiler Demystified	@samlee
From Embedded to Standalone: A Newcomer’s Transition to Hallo Flutter App Development	@cherry
メルカリハロのデザインシステムとFlutter	@atsumo
New Production Readiness Check experience in Mercari	@mshibuya
From Good to Great: Evolving Your Role as a Quality Consultant	@Udit
メルカリハロのプッシュ通知と CRM integration の話（Android編）	@sintario_2nd
LLMs at Work: Outsourcing External Service Review Grunt Work to AI	@danny, simon
メルカリ Tech Radarの取り組み	@motokiee
GitHubのBranch Protectionの突破方法	@iso
ナレッジマネジメントへの挑戦	@raven
メルカリハロにおけるFlutterアプリのQA戦略：クロスプラットフォーム開発のメリットと注意点	@um
mSCPとJamf Pro APIによるmacOSセキュリティ設定の手動IaC化の試行	@yu
Flutter Forward: Crafting Type-Safe Native Interfaces with Pigeons	@howie
メルカリハロのFlutter開発とSRE	@naka
Good tools are rare. We should make more!	@klausa
A smooth CDN provider migration and future initiatives	@hatappi
How to unit-test Mercari Hallo Flutter app	@Heejoon
Spanner Data Boostを活用したリアルタイムなリコンサイルエラーの検出	@yuki_watanabe
メルカリのEngineering Roadmapの具体的な運用について	@kimuras

Please bookmark this article and check it out when you want to read it so you can be aware of article publication notifications!

We’re looking forward to bringing you some interesting technology stories in the last month of 2024! I hope you’re looking forward to the Advent Calendar!

Designing a Zero Downtime Migration Solution with Strong Data Consistency – Part V

Wed, 13 Nov 2024 11:30:51 GMT

In the previous part, we covered how we are going to execute dual-write reliably. In this final part, we’ll discuss architecture transitions, rollback plans, and the overall migration steps. I hope this post provides valuable insights about how we achieve reversible actions at each phase.

Part I: Background of the migration and current state of the balance service
Part II: Challenges of the migration and my approach to address them
Part III: Mappings of the endpoints and the schema, client endpoint switches
Part IV: How to execute dual-write reliably
Part V: Architecture transitions, rollback plans, and the overall migration steps (this article)

Development Tasks

Here, I’d like to discuss the development tasks required to transition to the post-dual-write state. The topics we will cover include:

v1 batch applications, including accounting event processing
Accounting code processing
Historical data processing
Switching database client in bookkeeping service
Rewriting queries for BigQuery

Let’s begin with v1 batch applications. While I have previously covered the endpoint mappings between v1 and v2 APIs, I have not yet explained the mappings of batch applications. Currently, we have three kinds of v1 batch applications:

Batch applications with v1-specific logic, which can be further categorized into:
- Those based on business requirements, like the point expiration batch
- Those that don’t depend on business requirements, like the v1 data inconsistency validation batch
Batch applications without v1-specific logic, which are ad-hoc batch applications created for specific incidents

We won’t need to migrate batch applications that don’t have v1-specific logic. However, for those that do include v1-specific logic—regardless of whether they’re tied to business requirements or not—we need to create equivalent batch applications on the v2 side.

As I mentioned in the Accounting Event Processing section in Part I, we’ll still need to interact with the accounting service for event processing after dual-write is finished. Since the accounting event-related APIs guarantee idempotency, we’ll develop a batch application on v2 that replicates the logic of the existing v1 batches for sending and reconciling accounting events. During the transition, both batches will run in parallel. Once we’re nearing the completion of dual-write, we’ll phase out the v1 batch and ensure that all accounting events are successfully processed by the accounting service through reconciliation using just the v2 batch.

Now, regarding accounting code processing, the v1 balance service will continue to handle these even after dual-write is completed. To ensure backward compatibility, the v2 balance service will need to read from the v1 schema.

When it comes to processing historical data, we’re aware that it has developed without a well-defined ownership structure, and we plan to re-architect this area soon. As we move through this transition, we’ll need to modify how we write historical data during and after the dual-write phase.

In particular, the v1 balance service will be dedicated solely to reading historical data, while the v2 balance service will take over all write operations once the dual-write process is concluded. Now, let’s take a closer look at how the v2 balance service will manage the writing process for historical data.

While the accounting service ensures idempotency for processing accounting events, this guarantee does not apply to historical data managed by the v1 schema. Unfortunately, we can’t read results after a write operation, nor can we insert the same record multiple times within the same database transaction using mutations (for more details, please see the later Spanner Mutation Count Estimation section). As a result, when we finish the dual-write execution, we’ll need to implement the logic for inserting historical data from the v2 balance service into the v1 schema. At other times, the v1 balance service will take care of inserting historical data.

For the bookkeeping service, which currently connects directly to the v1 balance database, we’ll need to update its logic after the data backfill and before we complete the dual-write phase. This change will enable us to switch its single source of truth (SSOT) from the v1 schema to the v2 schema.

As for BigQuery, we’ll need to update all existing queries to focus exclusively on v2 data after the data backfill is complete. Considering that there are over 500 queries to modify, this task will take some time, so we will start it even before beginning the dual-write phase.

The following diagrams illustrate these changes:

Arrow A becomes A’, representing the revised logic for sending accounting events.
Arrow B becomes B’, indicating the updated reconciliation process for accounting events.
Arrow C becomes C’, signifying the bookkeeping service’s transition from the v1 schema to the v2 schema.
Arrow D marks the moment when we stop the dual-write logic.
Arrow E shows that the v2 balance service will start reading accounting codes from the v1 schema while simultaneously inserting historical data into the v1 schema.

Fig. 25: Architecture during dual-write phase

The following figure illustrates the final architecture once the dual-write process is complete:

Fig. 26: Final architecture after completing dual-write phase

Rollback Plans

Let’s describe the transitions of the architecture from figures A to E below, while addressing the availability of rolling back at each stage.

Transition from Phase A to Phase C (Request Proxy Phase)

In this transition, we can roll back without any additional effort since v1 requests will continue to be processed by the v1 balance service, aided by the request proxy implemented on the v2 balance service.

Transition from Phase C to Phase D (Dual-Write Phase)

Rolling back from the dual-write phase to the pre-dual-write phase would require us to remove any migrated data in the v2 schema. After the rollback, this data would no longer receive updates. When we resume the dual-write process, the latest data would need to be selected and replicated from the v1 schema to the v2 schema. In other words, if we don’t remove the outdated data from the v2 schema, subsequent requests could be processed based on this outdated data, potentially leading to errors or, worse, successful processing that results in data inconsistencies.

While it is safe to remove the migrated data from the v2 schema, we should have a mechanism in place to ensure that this data can be removed safely and efficiently.

Transition from Phase D’’ to Phase E (Post Dual-Write Phase)

Once we transition to the post-dual-write phase, rolling back will no longer be an option. Executing a rollback at this stage would require downtime, as the data in the v1 schema will become outdated soon after completing the dual-write.

Therefore, we must allocate time for synchronization to update the outdated v1 data with the latest information from the v2 schema. Only after this synchronization can a rollback be executed, if necessary.

Fig. 27: Initial state while developing the request proxy logic on the v2 balance service (A)

Fig. 28: Write client endpoint switch while initiating the request proxy (B)

Fig. 29: State when proxying requests (C)

Fig. 30: State during dual-write operations (D)

Fig. 31: State during dual-write operations and data backfill (D’)

Fig. 32: State before completing the dual-write (D’’)

Fig. 33: Final state after the dual-write process (E)

Spanner Mutation Count Estimation

When using Cloud Spanner, one key aspect we need to consider is the concept of mutation and its upper limit count.

Let’s visit the definition of mutation:

A mutation represents a sequence of inserts, updates, and deletes that Spanner applies atomically to different rows and tables in a database. You can include operations that apply to different rows, or different tables, in a mutation. After you define one or more mutations that contain one or more writes, you must apply the mutation to commit the write(s). Each change is applied in the order in which they were added to the mutation.

https://cloud.google.com/spanner/docs/dml-versus-mutations#mutations-concept

In Cloud Spanner, a mutation refers to the amount of data that will be affected in a single database transaction, quantified by a value calculated by Spanner. Although there is no specific formula for counting mutations, the documentation provides guidelines on how to count them for each insert, update, and delete operation.

Initially, Cloud Spanner supported a maximum of 20,000 mutations per database transaction. During that time, we faced significant challenges in avoiding the “Mutation limit exceeded” error. Fortunately, this limit increased to 40,000 and has now been raised to 80,000, alleviating our concerns about exceeding the limit in our processes.

With a dual-write solution, in general, we would be executing approximately twice as many database operations compared to those performed on either the v1 schema or the v2 schema. This will lead to a significantly higher total mutation count. As a result, it’s important for us to monitor the mutation count closely, particularly during dual-write operations, to ensure that we remain within the limit.

We have two options for measuring these counts:

Measuring them using the Go Spanner library
Estimating them based on database operations for each logic pathway

I would like to utilize both methods for measuring mutations. When measuring mutations using the library, we will need to prepare all the necessary test data to execute a specific logic path in the API. During the design phase, I dedicated one or two days to estimating mutation counts for all mappings of v1 and v2 APIs.

To estimate the mutation counts, I used formulas that incorporated variables representing the number of affected rows in specific tables. Since each API can have multiple execution paths, I focused on the paths that seemed most likely to result in the highest mutation counts.

To illustrate this process, let me provide a simplified example for easier understanding.

Consider an API called AuthorizeBalance, where user balances are represented as sums of individual BalanceComponents. For example, user A has a total balance of 200, consisting of four components: 100 + 50 + 30 + 20.

Now, if we update the Amount column in 1 row of the CustomerBalances table (which has 10 columns) and the Amount column in 4 rows of the CustomerBalanceComponents table (which has 15 columns), the initial mutation count could be calculated as 1 + 4 * 1 = 5. However, it’s important to highlight that when we perform these updates, we actually modify all columns—not just the ones being changed, but also any other columns that were selected during the read operations prior to the write.

In this case, we have:

Mutation count = 10 + 4 * 15 = 70

In reality, the total number of mutations could be significantly higher due to additional insertions and updates. Furthermore, as I explained in the example with just four balance components, the number of affected records can vary from user to user. Therefore, I represented this as a variable in the formula:

Mutation count = 10 + CustomerBalanceComponents * 15

With this formula, we can calculate the total mutation counts by substituting a specific number into the variable. I also analyzed how many rows could realistically be assigned to these variables based on results obtained in BigQuery. By querying how many resources were involved in a single request, I calculated the total mutation counts for each mapping and summarized how high they could be during dual-write execution. Fortunately, based on my estimation, the probability of exceeding the mutation count limit is nearly 0%.

Migration Steps

Let me summarize what we have discussed so far by presenting the migration steps as follows.

Bottom layer: The lowest square arrow represents each phase of the migration.
Second layer: The layer above indicates the transition when the read and write v1 balance clients switch their endpoints to v2.
Third layer: This layer represents when data backfill and data inconsistency check batch will be running.
Fourth layer: This layer details the execution of quality assurance (QA) before commencing the new phase.
Top layer: The topmost squared ovals encompass all development tasks necessary to transition to the subsequent phases.

One important thing to consider is how we approach this migration project as a whole. As we looked into the rollback options for each phase, we found that, in theory, we can move to the next phase and still be able to roll back to the previous one without major issues, except for the final rollback from the post dual-write phase. However, to be more cautious, we can first validate the entire migration process in a proof of concept (PoC) environment. Once we’ve validated everything there, we can follow the same procedures in the production environment.

The strong benefit of starting the migration in a PoC environment is that it allows us to make progress gradually. Therefore, I’d like to adopt this approach.

Fig. 34: Rough migration steps

Future Work

We have several tasks to complete before we can move forward with this migration. However, we currently have higher-priority work and are understaffed (we’re hiring!).

Given this situation, we’ll start with the pre-migration tasks when we can.

Key Takeaways

1. Focus on Minimal Goals

The saying "Those who chase two hares will catch neither" aptly describes the scale of this project. By minimizing the scope early and keeping it smaller, we increase our chances of success. External factors could disrupt the migration, necessitating additional fixes until completion. Thus, focusing our goals to the bare minimum is essential.

2. Importance of Research

At the outset of the project, I had no specific knowledge about system and data migration. However, after reading blog posts and articles, I’ve gained valuable insights into best practices and various perspectives that need to be considered.

3. Value of Thorough Investigations

We conducted a detailed investigation of the specifications for the v1 balance service. This investigation was crucial in designing a clear, well-informed solution. Even if the migration does not go as planned, the insights gained will be invaluable for managing the services.

4. Understanding the Details Accurately

Given the scale and complexity of this project, even small details matter. One minor misunderstanding can lead to disastrous consequences. That’s why I focused on following logic accurately, especially when new insights were provided by colleagues for each topic.

5. Evaluating Options and Trade-offs

Exploring various solutions and their trade-offs is essential, especially when preparing for unexpected situations. This approach helps identify critical issues and design the most suitable solutions.

6. Taking Calculated Risks

System and data migration is a substantial project, with some degree of risk being unavoidable. However, by breaking down the issues into manageable units, we can minimize these risks. For example, I estimated the Spanner Mutation counts for all v1 and v2 endpoint mappings.

7. Considering Reversible and Irreversible Actions

As we proceed, we must consider the rollback steps for every action. This is crucial for system and data migration, where an easy rollback process is essential for addressing issues. If we identify some irreversible actions during the design phase, those options may not be feasible or will require more careful consideration.

8. Example-Driven Communications

System and data migration is complex. Therefore, architects must provide clear and detailed diagrams to ensure other engineers understand the concepts without ambiguity.

Conclusion

In this series of posts, I have outlined the background of the migration and explained how I designed the solution for the system and data migration. I hope this information serves as a valuable reference for anyone considering various types of system and data migration.

Thanks for reading this far. Lead the future with these insights!

Designing a Zero Downtime Migration Solution with Strong Data Consistency – Part IV

Wed, 13 Nov 2024 11:29:03 GMT

In the previous part, we covered the mappings of the endpoints and the schema with client endpoint switches. In this part, we’ll discuss how to execute dual-write reliably. I hope this post provides valuable insights about how to design methods of online migration.

Part I: Background of the migration and current state of the balance service
Part II: Challenges of the migration and my approach to address them
Part III: Mappings of the endpoints and the schema, client endpoint switches
Part IV: How to execute dual-write reliably (this article)
Part V: Architecture transitions, rollback plans, and the overall migration steps

Dual-Write

Requirements

For online data migration, the functional requirement for dual-write is to support both reading and writing to v1 and v2 data. Specifically, a dual-write component will select both source and target data; if the target data does not exist, it will write the data to the target database. If it does exist, it will update the record.

The main non-functional requirement for dual-write is to minimize performance degradation, which is a challenge we need to tackle since some drop in performance is unavoidable when executing dual-write.

Dual-Write Component

Before we delve into the component responsible for executing dual-write, some readers may have questions about how we plan to implement it. This aspect will be detailed in the next Dual-Write Logic section, so for now, please assume that we can achieve dual-write through any suitable method.

Which component will execute the dual-write functionality? We have the following three options:

v1 balance service
v2 balance service
A new service

What if we consider using the v1 balance service as the component responsible for dual-write? It would work as follows:

Fig. 12: v1 balance service executes dual-write

At first glance, this approach seems reasonable. However, it actually introduces two types of race conditions as follows.

Fig. 13: Race Condition A – v1 clients switching their endpoints during dual-write

Fig. 14: Race Condition B – v1 clients switching their endpoints after completing dual-write

Race Condition A refers to a scenario where a CreateExchange request is processed on the v2 balance service before a CreateUserBalanceConsumption request is executed on the v1 balance service, both targeting the same balance account with an amount of 1000.

It’s important to note that CreateUserBalanceConsumption is a v1 API, in contrast to the CreateUserBalanceAddition logic discussed earlier, as this API deducts values from the credit side. Additionally, while the v2 CreateExchange API operates with double-entry bookkeeping, we will concentrate on the credit side for this explanation.

In this race condition, because the dual-write occurs from the v1 balance service to the v2 balance service (but not the other way around), any changes made on the v2 side won’t be reflected in the v1 data. As a result, the v1 balance service will detect a discrepancy between its data (Amount = 1000) and the v2 data (Amount = 0), ultimately leading to an inconsistent data error being returned to the client.

Race Condition B presents a variation of Race Condition A, where there is no dual-write involved. Even though the dual-write isn’t happening here, a similar situation can still arise. In this case, the consequences could be more severe than in Race Condition A, as the v1 balance service (which is supposed to handle the dual-write) would be unable to identify the differences between its data (Amount = 1000) and the v2 data (Amount = 0). This could allow the v1 CreateUserBalanceConsumption request to succeed, leading to further inconsistencies.

Could these race conditions occur in our environment? Yes, they can happen due to our canary deployment strategy, which allows us to test new images by deploying them as a single Kubernetes pod for a limited time. During this testing phase, some requests may be routed to the canary pod, while most requests will continue to be directed to the pods with the latest stable image.

What about the third option: using a new service? If we implement a new service that handles the dual-write instead of relying on the v1 and v2 services separately, the architecture would look like this:

Fig. 15: New service executes dual-write

With this option, client services would need to change their endpoints twice: first from v1 to the intermediate state (for the new service) and then again to v2. As mentioned earlier, we have two write clients and over 20 read clients, meaning the time required for all clients to make these endpoint changes would be considerable. Switching twice would take even longer due to high-priority tasks that may suddenly occupy the attention of those client service teams.

Considering all the options we’ve discussed, I believe the v2 balance service is the best fit for the dual-write component. However, we need to address one more important point regarding the timing of when v1 write clients should switch their endpoints to v2. Let’s explore this in more detail.

Race Condition C describes a situation similar to Race Condition A, with the primary difference being the direction of the dual-write (from the v2 balance service to the v1 balance service in Race Condition C). This means that similar issues could occur regardless of the choices made concerning the dual-write component.

FIg. 16: Race Condition C – v1 clients switching their endpoints when v2 balance service executes dual-write

As a result, v1 clients will need to switch their endpoints before executing the dual-write. This leads to a pre-transition period during which the v2 balance service internally calls the v1 endpoints for original v1 requests without executing any of the v2 logic. For more details, please refer to the upcoming Process Overview section.

Dual-Write Logic

In the previous section, I concluded that the v2 balance service is the most suitable choice for executing dual-write. In this section, I will discuss reliable methods for implementing dual-write, considering the following three options:

Google Cloud Datastream with Dataflow (CDC)
Single database transaction
Transactional outbox + worker

First, let’s examine the Google Cloud Datastream with Dataflow (CDC) approach. Google Cloud provides change data capture (CDC) through Datastream and data processing capabilities via Dataflow. Below are some important notes about Datastream, quoted from its documentation:

Question: How does Datastream handle uncommitted transactions in the database log files?

Answer: When database log files contain uncommitted transactions, if any transactions are rolled back, then the database reflects this in the log files as "reverse" data manipulation language (DML) operations. For example, a rolled-back INSERT operation will have a corresponding DELETE operation. Datastream reads these operations from the log files.

Question: Does Datastream guarantee ordering?

Answer: Although Datastream doesn’t guarantee ordering, it provides additional metadata for each event. This metadata can be used to ensure eventual consistency in the destination. Depending on the source, rate and frequency of changes, and other parameters, eventual consistency can generally be achieved within a 1-hour window.

https://cloud.google.com/datastream/docs/faq

Based on the above FAQ, Datastream supports only eventual consistency rather than strong consistency. Consequently, I concluded that it is not suitable for executing dual-write.

Next, let’s discuss the approach of utilizing a single database transaction for dual-write. By performing all database operations within a single database transaction, we can prevent any inconsistencies between the v1 schema and the v2 schema.

Fig. 17: Dual-write solution with single database transaction

Let’s revisit the non-functional requirements. Before we considered the single database transaction solution, our primary goal was to minimize API performance degradation. With the introduction of the Cloud Spanner database, we’ve identified an additional requirement, which can be summarized as follows:

Minimal API performance degradation
Compliance with the mutation count limit in Cloud Spanner

Regarding API latency, it’s clear that the v2 API latencies are likely to be worse than the current ones due to the extra database operations needed for the v1 schema. However, we’re uncertain about the degree of this degradation during the design phase. We’ll assess the performance metrics before moving forward with this approach.

The mutation count limit in Cloud Spanner refers to whether a single database transaction exceeds its allowed number of mutations, which is a specific term for the number of changes made within one transaction, with the limit set by Google Cloud. In other words, the more data we manipulate in one transaction, the more mutations we create, which can lead us to exceed the limit. If we surpass this limit, the transaction cannot be committed. We’ll address this topic in more detail in the dedicated Spanner Mutation Count Estimation section in Part V.

Finally, let’s consider the transactional outbox + worker approach. For a detailed explanation of the transactional outbox pattern, please refer to the documentation Pattern: Transactional outbox in microservice.io. In our case, its primary purpose is not to publish messages atomically, but to allow for atomic updates across different schemas.

In this approach, the v2 balance service reads the master data from the v1 schema and inserts a record as an asynchronous request into that schema. A newly introduced dual-write worker then retrieves this record and attempts to update the master data within the v1 schema. For this discussion, we will focus solely on the scenario after the v1 balance clients have successfully switched their endpoints, as concluded in the previous section.

Fig. 18: Dual-write solution with transactional outbox + worker

If we encounter the issues mentioned above, such as API performance degradation and/or exceeding the mutation count limit, it may be worthwhile to consider the transactional outbox + worker approach. This would allow us to reduce the number of database operations, helping to mitigate those issues. However, an important trade-off with this approach is that we must accept the possibility of inconsistent data between v1 and v2 as long as there are unprocessed asynchronous request records in the v1 schema.

Consequently, I would like to propose the single database transaction approach as a dual-write solution. The subsequent sections are written with this single database transaction solution in mind.

Process Overview

In this section, I will explain how the balance client handles requests and responses, as well as how the v2 balance service executes its logic and database operations in conjunction with the single database transaction dual-write solution.

To summarize, the following outlines the process. Important changes are indicated with underlined text.

Current state
- Proto interface
  - Request: v1
  - Response: v1
- Database
  - v1 balance service reads/writes only v1 data

This phase is consistent with the current state described in the Current State section in Part I. One important point to note is that the request proxy logic for the v2 balance service is developed in advance.

Fig. 19: State after introducing request proxy logic in v2 balance service

State while migrating v1 endpoints to v2
- Proto interface
  - Request: v2
  - Response: v2
- Database
  - v2 balance service reads/writes only v1 data for v1 requests

This phase describes the scenario where v1 balance clients switch their endpoints to v2 to call the v2 balance service APIs. With the request proxy logic implemented in the previous phase, v2 balance clients continue to manage their data in the v1 schema through the v2 balance service. At this stage, the request proxy logic invokes the v1 balance service logic to delegate the original processing and does not yet manipulate data in v2.

Starting from this phase, the client endpoint switch including any necessary mappings with wrapper APIs and the v1/v2 endpoint mappings will be applied, as the v2 balance service needs to accept v1 balance requests using v2 proto request interfaces while the v1 balance client must receive v1 balance responses through v2 proto response interfaces.

As previously mentioned, this phase is necessary to transition v1 balance endpoints to v2 without any significant impact, facilitating an easy rollback if needed. Even if some balance clients revert their endpoint switch, their data will have been managed solely by the v1 balance service logic, thereby avoiding any data consistency issues.

Fig. 20: State when v1 clients switch their endpoints from v1 to v2

State in dual-write
- Proto interface
  - Request: v2
  - Response: v2
- Database
- v2 balance service reads/writes both v1 and v2 data for v1 requests

This phase marks the beginning of dual-write functionality by the v2 balance service. After releasing the dual-write logic in the v2 balance service, it will start duplicating data from the v1 schema to the v2 schema based on the established v1/v2 schema mappings.

The v2 balance service attempts to fetch data from the v1 schema, and if the corresponding v1 data does not exist in the v2 schema, it will insert it there. If the data does already exist, the v2 balance service will read and update it.

Fig. 21: State after dual-write starts

Final state (after dual-write)
- Proto interface
  - Request: v2
  - Response: v2
- Database
  - v2 server reads/writes only v2 data for all requests

In this final phase, the v2 balance service completely transitions away from the dual-write logic and processes requests just as it did prior to this series of steps. At this stage, both v1 and v2 requests are managed seamlessly and without distinction.

Fig. 22: State after dual-write ends

Data Backfill

Data backfill refers to the migration of data from the source database to the destination database. In this context, it specifically involves the transfer of data from the v1 schema to the v2 schema.

Let’s consider the scenario without data backfill. For instance, if some users have used our payment functionalities prior to the implementation of dual-write and do not take any action during the dual-write phase, they may encounter a NotFound error when they later attempt to make a payment. This occurs because dual-write has not replicated the users’ data to the v2 schema, resulting in no corresponding data being available in v2 at that time. Therefore, executing data backfill is essential for a successful system migration.

An important requirement for data backfill is to address existing inconsistent data. This presents a valuable opportunity to identify critical inconsistencies that we may not have previously detected. We must enforce this requirement, as we assume that the invariance verification batch, which I will explain in the next section, will run for both the v1 and v2 schemas before initiating dual-write. However, it is possible that we might inadvertently migrate inconsistent data to the destination database, and I would consider this option in the future if necessary. Moreover, since the v2 balance service will continue referencing v1 data, we would need to address increased database load and latencies that may occur during the data backfill period.

The total number of records to be migrated to the v2 schema could be up to hundreds of billions, raising the question of how to reduce the volume of data backfill. Fortunately, dual-write can significantly reduce the need for data backfill, as it replicates v1 data to the v2 schema in real-time. We can benefit the most by performing the data backfill after running dual-write for a while, because by that point, we hope that most active user data will have already been migrated to the v2 schema.

We should execute data backfill during the dual-write phase rather than at other times for the following reasons:

If we execute data backfill before dual-write, some migrated data could become outdated when we start dual-write, as the migrated data would not be updated thereafter
If we execute data backfill after dual-write, the source data would likely be outdated since it would not be updated after finishing dual-write

We will execute data backfill using a dedicated batch application. Given that both the v1 and v2 schema reside in the same database, the batch application will perform the following operations for each identical pair of resources within a single database transaction:

Select the v1 resource
Select the v2 resource
If the v2 resource exists, do nothing (as it has already been replicated via dual-write)
Otherwise, insert the identical data into the v2 schema

Note: In the following figure, both the v1 and v2 schemas actually reside within the same database; however, they are depicted as separate databases for the sake of clarity and ease of understanding.

Fig. 23: Data backfill

When considering which data to backfill, it is easier to identify the data that will not be backfilled. While I will elaborate on this in the Development Tasks section in Part V, some data will remain untransferred since the v1 balance service will continue to manage it even after dual-write is complete. Conversely, we will definitely backfill the v1 master data, also known as v1 resource data.

For non-master data, such as logs and snapshots of specific resources, the decision to backfill depends on whether the v2 balance service logic references this data. If there are no corresponding records in v2, the v2 logic may not function properly.

More specifically, if no requests are made during dual-write for certain data (meaning it isn’t migrated to the v2 schema via dual-write), the v2 balance service may successfully locate the master data migrated from v1 through backfill, but it may not find the dependent data, such as logs and snapshots. If any v2 logic relies on this non-master data, the v2 balance service could return a data loss error or an inconsistent data error due to the absence of those records.

I plan to revisit this point in the future to clarify the exact targets for backfill.

We will consider continuing dual-write for a longer period than initially planned as a fallback option for the future. This is based on the premise that all source data will eventually be migrated to the destination database, as long as we theoretically maintain the execution of dual-write.

Another option is to forgo both dual-write and data backfill, allowing the v1 data to remain in the v1 schema. It’s important to note that this differs from continuing dual-write; both options would not involve data backfill, but the distinction lies in whether we persist in executing dual-write. Specifically, this option indicates that the v2 balance service does not replicate v1 data to v2, but instead manages the v1 data directly.

I’ve considered this approach because it has the advantage of eliminating the need for data migration. If we opt for this path, the situation would be as follows:

v1 balance clients switch their endpoints to v2
The v2 balance service manages v1 data for v1 requests via the v1 balance service, while handling v2 requests for v2 data

In this scenario, we would have migrated only the client endpoints from v1 to v2, while each service logic would continue to operate in its original location. This means that the v1 balance service and the v2 balance service would operate independently rather than interchangeably. As a result, the v1 balance service logic and data would still reside in v1, which means we would not leverage the migration. Additionally, we might still need to address any issues with the v1 logic if they arise, ultimately not reducing total costs.

Fig. 24: No data backfill, and v2 balance service handles requests respectively

Data Inconsistency Check

Using a single database transaction helps us minimize the risk of inconsistent data that could be introduced by dual-write operations. However, in the event that inconsistencies do occur, it is essential that we detect and resolve them as quickly as possible. To achieve this, we will develop a batch application that verifies the consistency of the data using Cloud Spanner’s ReadOnlyTransaction, which does not lock any rows or tables. I won’t go into the specifics of each consistency check here.

When verifying the consistency of bulk data, one important aspect is ensuring that the data is consistent at a specific point in time. I initially considered using BigQuery, which replicates data from our production databases. However, I realized that we cannot completely avoid inconsistent data because each table is replicated on its own schedules.

There are three types of inconsistent data:

Inconsistencies within the v1 schema
Inconsistencies within the v2 schema
Inconsistencies between the v1 and v2 schema

The first two types are relatively straightforward; for instance, the Amount value in the Accounts table should match the corresponding value in the latest AccountSnapshots table at the same point in time. The third type, on the other hand, is more complex.

It’s important to note that we will be matching the primary keys of v1 resources with those of v2 resources. Fortunately, since both v1 and v2 data reside in the same Spanner database, we can take advantage of this setup by selecting and comparing both resource types in a single query. While the schemas differ, there are certain consistencies between them that we will verify through the batch application.

Furthermore, we will ensure that the results of each read and write operation for both the v1 and v2 databases are identical during the dual-write process. Although this approach is more ad-hoc, it is essential for facilitating immediate verification without having to wait for the next execution of the data inconsistency check batch process.

In this part, we covered how we are going to execute dual-write reliably. In the final Part V, we’ll discuss architecture transitions, rollback plans, and the overall migration steps.

Designing a Zero Downtime Migration Solution with Strong Data Consistency – Part III

Wed, 13 Nov 2024 11:27:17 GMT

In the previous part, we covered the challenges of the migration and my approach to address them. In this part, we’ll discuss the mappings of the endpoints and the schema with endpoint switches on client sides.

Part I: Background of the migration and current state of the balance service
Part II: Challenges of the migration and my approach to address them
Part III: Mappings of the endpoints and the schema, client endpoint switches (this article)
Part IV: How to execute dual-write reliably
Part V: Architecture transitions, rollback plans, and the overall migration steps

Client Endpoint Switch

Let’s begin with the client endpoint switch.

There are only two write clients for the v1 balance. However, the number of v1 read clients has grown to over 20, with many client services directly calling specific v1 balance APIs.

To reduce the time and cost associated with this switching process, I’ve considered grouping multiple calls to the same v1 balance API under one wrapper service call.

For example, let’s say there are five client services that call the v1 balance GetX API, and one of these services provides a wrapper API for the v1 GetX API. This wrapper API Internally calls the v1 GetX API and returns its response to the caller. In this scenario, we could switch the endpoints for all client services, except for the one providing the wrapper API, from the v1 balance to the wrapper client. See the following figure, which visualizes this transition:

Fig. 10: Switching endpoints from v1 balance to the wrapper client

With this approach, the number of endpoint switches will be reduced from five (for clients A to E) to just one (for client C) when switching from v1 to v2.

However, we need to dig deep into the following point more:

Whether or not there is a wrapper API that can accept all types of request parameters specified by other clients and return all types of response parameters utilized by them
- If not, whether the client service team has available resources to develop it

v1/v2 Endpoint Mappings

After the migration, only the v2 API will remain active, while the v1 API will basically cease processing requests. Therefore, I summarized the mappings between v1 APIs and v2 APIs.

I organized these mappings into four types:

v1 APIs mapped to existing v2 APIs
v1 APIs mapped to new v2 APIs
Unmapped v1 APIs
Unmapped v2 APIs

The first type comprises the actual mappings of existing v1 APIs to their corresponding v2 APIs, while the second type refers to new mappings involving v1 APIs and new v2 APIs that will be developed in the future.

The last two types merit further discussion. Unmapped v1 APIs indicate those that will not be migrated to v2. I will elaborate on this later, but it’s important to note that some v1 APIs will indeed not be migrated. Unmapped v2 APIs represent newly introduced functionalities in v2; hence, there are no corresponding candidates in the v1 APIs.

As noted in the Current State section in Part I, the v1 API operates on a single-entry bookkeeping model, while the v2 API utilizes a double-entry bookkeeping approach. In other words, the v1 balance service supports only credit or debit transactions, while the v2 balance service can handle both. This raises a critical question: how do we address the missing side of the double-entry bookkeeping data in v2 when migrating from v1?

Thus far, I haven’t delved into the specifics of the v1 and v2 APIs. To better understand the technical issues at hand, let’s examine some details.

The v1 CreateUserBalanceAddition API is used to grant a set of values to a user (or partner), essentially functioning as a debit operation in double-entry bookkeeping. Clients can specify M AdditionMethods (debit) to indicate the types of values being granted, such as funds and/or points. The equivalent v2 CreateExchange API requires clients to specify N Source (credit) information and one Target (debit) information.

However, the v1 CreateUserBalanceAddition API client cannot specify the credit side in the v2 CreateExchange request parameters because that information is not passed along by upstream services (recall that the v1 CreateUserBalanceAddition API only accepts debit information). As a result, they will have to use dummy values.

While the v1 CreateUserBalanceAddition allows for M AdditionMethods, the v2 CreateExchange is limited to N Source and 1 Target. If we map CreateUserBalanceAddition to CreateExchange, the M AdditionMethods would only map to 1 Target, which means CreateExchange cannot accept multiple AdditionMethod inputs.

Considering the available options to resolve this problem and their trade-offs, I advocate for enhancing CreateExchange to accept multiple Target information. By implementing this change, M AdditionMethods could be mapped directly to M Target entries, allowing the write client to maintain its current implementation with minimal adjustments.

We will continue to communicate with the payment service (write client) team to explore further solutions to this issue.

Fig. 11: Summary of CreateUserBalanceAddition and CreateExchange

After the migration, most requests to the v2 balance service—except those that were originally made directly to the v2 balance service without switching endpoints—will involve either credit or debit information, as outlined in the mapping above. Future migration tasks will include a step to consolidate multiple single-entry bookkeeping requests into a single double-entry bookkeeping request, which will require the write client (payment service) to adjust its logic accordingly.

Similar to the accounting service migration described in the Alignment section in Part II, this task was considered but ultimately excluded from the scope. It would require considerable effort, especially because of the breaking changes involved in how accounting events are sent and reconciled after being converted into double-entry bookkeeping data.

v1/v2 Schema Mappings

As discussed in the v1/v2 Endpoint Mappings section, I have also organized the mappings between the v1 schema and the v2 schema into four types, similar to those in the endpoint mappings:

v1 tables/columns mapped to existing v2 tables
v1 tables/columns mapped to new v2 tables
Unmapped v1 tables
Unmapped v2 tables

An important note here is the need to match the primary keys (PKs) of v1 resources with those of v2 resources. Although I will explain the rationale behind this requirement later, adopting this policy will facilitate a smoother migration process.

In this article, we covered the mappings of the endpoints and the schema with client endpoint switches. In Part IV, we’ll discuss how to execute dual-write reliably.

Designing a Zero Downtime Migration Solution with Strong Data Consistency – Part II

Wed, 13 Nov 2024 11:25:23 GMT

In the previous part, we covered the background of the migration and the current state of the balance service. In this part, we’ll discuss the challenges of the migration and my proposed approach to addressing them. I hope this post provides valuable insights about how to prepare for a massive migration project.

Part I: Background of the migration and current state of the balance service
Part II: Challenges of the migration and my approach to address them (this article)
Part III: Mappings of the endpoints and the schema, client endpoint switches
Part IV: How to execute dual-write reliably
Part V: Architecture transitions, rollback plans, and the overall migration steps

Challenges

We face several requirements during the migration, which include:

Zero downtime
No data loss
Strong data consistency (i.e., no eventual consistency)
Availability
Performance
Reliability (ensuring that no bugs are introduced)

The most challenging constraint is zero downtime, which prompts us to consider an online migration approach. However, adhering to other constraints makes the entire migration process significantly more complex than it would be if we were able to compromise on some of them.

As previously discussed, the v1 balance service has the following dependencies:

Accounting event processing
Accounting code processing
Historical data processing
Bookkeeping (which directly connects to the v1 balance database)
BigQuery (for querying v1 data)

More specifically, even during the migration, we need to ensure the following:

Continued sending and reconciling of accounting events to the accounting service
Ongoing reading and writing of accounting codes
Continuous reading and writing of historical data
Ensuring the bookkeeping service can execute its logic using up-to-date balance data
Guaranteeing that each query reads up-to-date balance data

Additionally, we must address the following concerns:

What range of data needs to be migrated
- Only specific data, which may require v1 data as a complete dataset
- All data
The timing and method by which read/write v1 balance clients will switch their endpoints to v2
- How read/write v1 balance clients will handle mixed logic for both v1 and v2 API calls
- How read/write v1 balance clients will be informed about the version in which their data exists
The ease of rolling back individual migration phases or even the entire migration after migrating certain v1 behaviors and their corresponding data to v2

These are not all of our challenges. An additional implicit challenge looms: the ongoing changes happening in both systems until we complete the migration.

What if we need to update the v1 schema in the midst of the data migration? Any changes made to the v1 schema will also have to be reflected in the v2 schema. Otherwise, even after completing the migration, some behaviors or data may be lost.

In essence, the longer the migration period, the more we need to migrate. This is particularly significant for a large-scale migration project like ours. We essentially need to track the types of behaviors and/or data introduced to the v1 system until we finish the migration. As you can imagine, this will be a substantial effort.

Approach

I’ve covered all assumptions for the migration while providing an overview of the system so far. Now, let’s dive into our migration approach.

Learning Best Practices

We don’t need to reinvent the wheel from scratch. Before diving into the design, I focused on learning the best practices for both system and data migration by reading over 80 articles. This gave me a comprehensive understanding of the migration process, including common approaches like online migration and typical pitfalls to watch out for as follows:

Whether each phase can be rolled back
Strong consistency or eventual consistency
Inconsistent data
How clients know where their data is located

For a list of the articles I read, please see the References section at the end of this post.

Migration Roadmap

How many months or years will this work require? I couldn’t answer this question with reasonable accuracy at the beginning of the project, but I can provide a more informed estimate now that I have developed a migration roadmap and a design doc.

Early in the project, I created a migration task list that outlines a range of specific tasks, presented as bullet points, which must be completed throughout the migration process. There are two main reasons for creating this list:

To identify essential tasks for the migration
To understand the scale of the migration based on those tasks

With insights gained from best practices in system and data migration, I was able to identify the necessary tasks for the entire migration, even before designing the solution. All tasks identified are listed below; however, it’s important to note that I have not yet completed all the tasks in phase 1.

Phase 1. Investigation
- Assess migration feasibility
  - Determine API migration granularity
  - Investigate compatibility between v1 and v2 APIs
  - Implement new v2 APIs
  - Check existing database logic such as stored procedures, triggers, and views
  - Verify compatibility between v1 and v2 schema/data models
  - Validate compatibility between v1 and v2 batch applications
  - Review PubSub-related logic
  - Identify dependent services
  - Identify deprecated v1 APIs
  - Read and understand v1 API code
  - Investigate and resolve issues
- Clarify dependencies
  - Application dependencies
    - Go version
    - Library/package version
    - Environment variables
    - Estimate Spanner mutation limit
  - Assess network limitations
    - Allowed ingress/egress namespaces
  - Review IAM/privilege limitations
    - Request validations
  - Upstream services analysis
    - Review v1 request parameters
    - Review v1 response parameters
    - Identify v1 API use cases
  - Evaluate subscribed topic/message (PubSub)
  - Downstream services analysis
  - Infrastructure
  - Environment setup
    - sandbox
    - test
  - DB clients
    - Bookkeeping service
  - Manual operations (e.g., queries for BigQuery)
  - Monitoring setup
    - SLOs
    - Availability
  - Tools
    - Slack Bot
    - CI
      - GitHub Actions
      - CI software
    - Linter (golangci-lint)
  - Stakeholder identification
    - Payment team
    - Accounting team
    - Compliance team
  - Compliance adherence
    - JSOX
- Documentation
  - Design document
  - v1 change log
  - v1 inventory
  - Migration schedule
  - Criteria for deleting PoC and production v1 environments
  - Cloud cost estimation
  - Risk assessment
  - Production migration instructions
  - Post-migration operation manual
  - Technical debt summary
  - Upgrade task list
  - QA test instructions
  - Rollback test instructions
  - Operation test instructions
  - Data backfill test instructions
  - Performance test instructions
  - Client team onboarding document
  - Balance team onboarding document
  - v2 playbooks for each alert
Phase 2. PoC
- Set up PoC environment
- Fix balance service
  - Update v2 proto interface
  - Implement request proxy logic
  - Develop data consistency validation batch
  - Migrate v1 test code to v2
- Fix client logic
- Set up tools
  - Datadog dashboard
- Conduct QA
- Conduct performance tests
- Conduct rollback tests
- Conduct operation tests
- Conduct tool tests
- Conduct data backfill tests
- Monitor data migration, performance, and Spanner mutation count
Phase 3. Migration on production environment
- Switch client endpoints
- Set up monitoring
- Fix v1 data to pass data consistency checks
- Perform data backfill
- Monitor data migration, performance, and Spanner mutation count
- Backup data
- Discontinue PoC environment
- Discontinue production environment

Furthermore, I organized these tasks by their dependencies and created a roadmap to provide a rough timeline. I provided estimates based on my experience, though I acknowledge that my estimates may not be entirely reliable. Ultimately, this process indicated that the overall timeline could range from two to four years. However, this estimate lacks precision due to the absence of a detailed design and additional supporting resources.

Fig. 8: Roadmap based on the migration task list

In our case, we didn’t need to provide a strict estimate for the schedule at the start of the project. If you’re required to estimate the overall timeline, you can create a roadmap as described above. Once you prepare a design document, you can then refine and support each estimate based on the detailed design.

I admit this is not the most polished format for a migration roadmap. However, I believe it works effectively for estimating the schedule, identifying dependencies, and designing a solution for the migration.

Investigations

With significant assistance from @mosakapi, we gathered almost all the necessary information on the following topics:

The request/response parameter mappings between v1 and v2 APIs
The schema mappings between v1 and v2 tables
The locations where v1 APIs are invoked by all read/write clients
v1 API specifications
v1 batch specifications
Dependent services
PubSub messages and their subscribers
Spanner DB clients (bookkeeping service)
Queries for v1 data (BigQuery)

Since the v2 balance service was released in February of this year and is still relatively new, we were able to collect information about the v2 specifications efficiently, without consuming a significant amount of time.

Alignment

Before designing the solution, I reviewed documents outlining the future roadmap of the payment platform to which my team belongs. It is essential to align the post-migration architecture with the vision described in the future roadmap.

However, it’s also important to acknowledge that we cannot achieve the architecture described in the future roadmap through a single, comprehensive system migration. Therefore, as we proceed with any type of migration, we need to clearly define the migration scope and plan for the subsequent steps following the initial migration.

In fact, we have a roadmap for migrating the accounting service to a newer version, as outlined in the future roadmap document. Initially, I included this migration in the project’s goals. However, I’ve come to realize that completing the accounting system migration in this phase is not feasible due to the additional effort and timeline required. The migration involves extra tasks, such as replicating the functionalities currently offered by the existing accounting service in the new version and ensuring their reliability and performance.

Design Direction

Are you familiar with the book Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith? It’s an excellent resource. The book advocates for the Strangler Fig application pattern, where developers gradually break down a large monolithic application into smaller microservices.

We initially considered this approach as the foundation for our migration, intending to migrate smaller parts of v1 behaviors and data into v2 one by one. However, during the design process, I discovered that this gradual migration strategy could be significantly challenging with our dependencies and concerns outlined in the earlier Challenges section.

Take a look at the figure below, which illustrates the API dependency graph. Some APIs are used exclusively by specific resources, while others are accessed by many resources. There are also loosely grouped API suites called by certain sets of resources. However, this loose grouping—with some APIs being accessed by other resources—makes it challenging to gradually migrate smaller parts of the v1 balance service.

Fig. 9: API dependency graph

To be honest, designing a gradual migration plan while considering these dependencies and concerns to resolve them properly would have taken me much longer than six months.

Therefore, I prioritized reversible actions over gradual migration, particularly regarding the ease of rollback. In some situations, rollback may be impossible, leading to potential downtime if we encounter issues. We can experiment with reversible actions more rapidly than with irreversible actions, allowing for quicker iterations through trial and error. In the following sections, I will explain the solution based on this principle.

As I mentioned in the Challenges section, the most critical constraint is achieving zero downtime while simultaneously managing other constraints. To address this, we plan to execute an online migration with data backfill, enabling us to migrate data without incurring any downtime. I will explain how we implement online migration while also addressing various other concerns. For more details, please refer to the Dual-Write section in Part IV.

In Part III, we’ll discuss the mappings of the endpoints and the schema with endpoint switches on client sides.

References

Designing a Zero Downtime Migration Solution with Strong Data Consistency – Part I

Wed, 13 Nov 2024 11:00:09 GMT

At our company, we have a payment platform that provides various payment functionalities for our users. One key component of this platform is a balance microservice that currently operates in two versions: v1 and v2.

The v1 balance service is designed as a single-entry bookkeeping system, while v2 is designed as a double-entry bookkeeping system. Although there is currently no direct compatibility between v1 and v2, achieving compatibility is not impossible.

Over the past six months, we’ve been investigating how to migrate from the v1 service to the v2 service. The main reason for this migration is that v2 is built with more modern and organized code, which could significantly reduce development costs when fixing bugs and adding new features.

Another motivation for using the newer version of the balance service (v2) lies in the power of double-entry bookkeeping. One key aspect of double-entry bookkeeping is its ability to handle two sets of accounting data as a single transaction: credit (the provision side) and debit (the receiving side). In contrast, single-entry bookkeeping only allows us to track one side of a transaction, which can leave us uncertain about the source or target of that transaction. However, double-entry bookkeeping provides a complete view, enabling us to validate whether the combinations of credit and debit are valid.

The goal of this migration is to transition nearly all functionalities from the v1 balance service to the v2 balance service. While we aim to migrate most features, we recognize that there may be exceptions where some functions might still need to be managed by the v1 balance service. The scope of the migration encompasses all components that are impacted by this transition.

Disclaimer:
Please note that we have NOT yet gone through the actual migration process. Also, the design might change after this series of posts goes live. Even without having experienced the migration process myself, I am publishing this series of posts because I believe I can contribute to the industry by offering valuable insights on considerations and design methods for system and data migrations, which can be quite massive in scale and significantly complex.

I will cover the following topics to give you a clearer understanding of our system and data migration solution:

Details of the solution we intend to execute
My design approach for the solution

What I won’t be discussing includes:

Our experiences with system migration
Proven best practices for system migration
Specific domain knowledge related to accounting, bookkeeping, and payment transactions

This blog is divided into 5 parts as follows:

Part I: Background of the migration and current state of the balance service (this article)
Part II: Challenges of the migration and my approach to address them
Part III: Mappings of the endpoints and the schema, client endpoint switches
Part IV: How to execute dual-write reliably
Part V: Architecture transitions, rollback plans, and the overall migration steps

I hope this series of posts provides valuable insights for anyone involved in migration projects.

Acknowledgments

I extend my heartfelt gratitude to @mosakapi, @foghost, and @susho for their invaluable assistance. Special thanks also go to all teams involved for their continuous support.

Current State

Let’s outline the tech stack and current architecture of the balance service first.

The tech stack is as follows:

Go
Kubernetes
gRPC (with protocol buffers)
Google Cloud Platform
- Cloud Spanner
- Cloud PubSub

Both v1 and v2 have their own gRPC services managed by a single Kubernetes deployment, which means they feature distinct APIs (proto interfaces) and batch applications. Additionally, we use canary deployments when deploying new images.

Also, they each have different database schemas (data models) managed by a single Cloud Spanner database. There are no (materialized) views, triggers, or stored procedures in either version.

The following figure illustrates the architecture more clearly:

Fig. 1: Two versions (v1 and v2) of the balance service

Then, let’s explore the architecture of components related to the balance service.

Accounting Event Processing

When Mercari awards points to users, we need to keep track of their addition, subtraction, expiration, and consumption. To handle this, we have a dedicated accounting microservice, while the v1 balance service delegates these accounting tasks to it.

Right now, the accounting service functions as a single-entry bookkeeping system, just like the v1 balance service. Client services must perform two key actions: sending accounting events and reconciling those events afterward. The accounting service supports a Pub/Sub system for sending events and an API for reconciliation. To ensure timely publication of accounting events, multiple services are involved in publishing/reconciling these events, and the payment service also sends and reconciles accounting events on its own.

Currently, the accounting team relies entirely on the accounting service for their operations. Therefore, even after we migrate to the new system, it’s essential that the v2 balance service continues to publish accounting events to the Pub/Sub topic and also handles reconciling those events.

Fig. 2: Architecture of the accounting service

Accounting Code Processing

Along with processing accounting events, there’s another internal concept related to accounting called “accounting code”. This is a string value that indicates the purpose of payment actions.

The payment service calls the v1 balance APIs using the accounting code, and the v1 balance service checks the validity of the request by verifying whether the specified accounting code exists in the balance database.

Registering a new accounting code can be done through Slack using a slash command. This command triggers a webhook to the Slack bot server, which then publishes messages for the accounting code registration, allowing the v1 balance service to subscribe to them and insert the specified code.

Additionally, the v1 balance service offers a GetAccountingCode API for GET requests, enabling client services to verify whether an accounting code exists before submitting their requests.

Fig. 3: Architecture related to accounting code

Historical Data Processing

The v1 balance service not only manages the latest values of user funds, points, and sales, but also maintains historical data for them.

When users initiate specific payment actions, the payment service calls the v1 balance APIs and includes relevant historical information as metadata. The v1 balance service processes this request and saves the provided metadata.

To access historical data, the v1 balance service offers GET APIs. When these APIs are called, they return a history entity along with the metadata in the response.

The history service uses these APIs to construct the finalized historical record based on the returned information and then provides it to the client. Additionally, they may call other service APIs to retrieve details about the original payment information.

Fig. 4: Architecture related to historical data

Bookkeeping

We have a bookkeeping service that functions as a legal ledger component and consists entirely of batch applications.

Ideally, each microservice should maintain its own database and access information from other services via API calls. However, since the bookkeeping process demands a significant amount of balance data, the bookkeeping service directly connects to the v1 balance database to carry out its operations most efficiently.

Fig. 5: Bookkeeping service

BigQuery

Certain business operations rely on queries against the v1 schema in BigQuery, meaning there are dependencies on v1 data managed by the v1 balance service. In fact, there are more than 500 queries that utilize this v1 data.

Fig. 6: BigQuery depending on v1 data

The following figure summarizes all the related components described so far, serving as a blueprint that I created for designing the solution. Please note that for convenience, I have split the v1 and v2 balance services and their databases (schemas) into two distinct components.

Fig. 7: Current components related to the v1 and v2 balance services

In this article, we covered the background of the migration and the current state of the balance service. In Part II, we’ll discuss challenges of the migration and my proposed approach to addressing them.

We hold mercari.go #27

Mon, 11 Nov 2024 13:22:50 GMT

Introduction

Hello, we are the mercari.go staff, kobaryo, and earlgray.

On September 19th, we hosted a Go study session called mercari.go #27 via a YouTube online broadcast. In this article, we’ll briefly introduce each presentation from that day. The videos have also been uploaded, so please look at them as well.

Writing profitable tests in Go

The first session was “Writing profitable tests in Go“ by @kinbiko.

Presentation material: Writing profitable tests in Go

The session introduced the theme of testing in Go from the perspective of profitability. In this session, @kinbiko introduced the rules for deciding whether to write tests and techniques for describing tests in Go. Tests are useful not only for verifying the behavior of code but also for ensuring that future changes do not cause issues.

However, tests cause costs in terms of writing time and execution, so it is important to justify these costs. You can do this by calculating the expected impact of the lack of a test, based on your organization’s historical incident impact and engineering salaries multiplied by the probability of needing to spend time on incident handling / debugging as a result of a missing test.

Additionally, tips were provided, such as the benefits of improving readability and code quality in Go tests, and the drawbacks of forcing the use of table-driven tests where separate subtests are more readable. Various other tips were introduced, so if you’re interested, please take a look. Table-driven tests are often seen in Go, and many people tend to write in this style. I was also one of them, but this time, I was able to understand their advantages and disadvantages, so I want to use them in appropriate use cases going forward. (earlgray)

GC24 Recap: Interface Internals

The second session was “GC24 Recap: Interface Internals” by @task4233.

Presentation material: GC24 Recap: Interface Internals

In this session, as a recap of the “Interface Internals” presentation at GopherCon 2024, the speaker explained how function calls through interfaces are executed, using a debugger to see values in memory.

When a Go program is compiled to assembly, we can understand that the function is invoked by a call instruction with an argument that is the memory address where the function’s processing is written. However, since a method call via an interface dynamically selects the function to be invoked, this mechanism cannot be used as is. This session started with explaining the data structures that implement interfaces, followed by the method to determine the address of the called method and techniques to speed up this process.

As the content of this presentation is deep and core part of the Go language, I personally felt the need to read references and watch it multiple times to properly understand it. (kobaryo)

GC24 Recap: Who Tests the Tests?

The third session was “GC24 Recap: Who Tests the Tests?” by @Ruslan.

Presentation material: GC24 Recap: Who Tests the Tests?

This session, like the second GC24 Recap: Interface Internals, was a recap of GopherCon 2024, covering the content of “Who Tests the Tests?”

We use test coverage as an indicator of software quality, but it does not guarantee the quality of the tests themselves. This session introduced Mutation Testing to ensure the quality of tests. By using this technique, it can be checked whether tests fail when operators or boolean values in a program are changed, ensuring that the tests only pass the correct program. Additionally, the method of automatically generating such programs using the AST package was explained.

The session provided fascinating content about ensuring the quality of the tests themselves, and it was highly practical, making it a very beneficial session. Readers of this blog might also consider introducing this technique. (kobaryo)

Cloud Pub/Sub – High Speed In-App Notification Delivery

The fourth session was “Cloud Pub/Sub – High Speed In-App Notification Delivery“ by @akram.

Presentation material: Cloud Pub/Sub – High Speed In-App Notification Delivery

A case study on the utilization of Cloud Pub/Sub in the Notification platform for managing notifications at Mercari was introduced. At Mercari, notifications such as in-app alerts, To-Do lists, emails, and Push notifications are sent to customers. To achieve performance that enables real-time and asynchronous notifications to over 20 million customers, the notification platform uses Cloud Pub/Sub. Specifically, the notification process is handled by a two-server configuration: one server receives Push notification requests and publishes them to Pub/Sub, and the other subscribes to Pub/Sub and performs the actual notifications. As a result, Mercari currently achieves more than 16 million Push notifications per day (400 rps at peak).

This was a very interesting insight into the use of Pub/Sub in a large-scale platform like Mercari. If you are experiencing performance challenges with handling asynchronous tasks, considering the introduction of Pub/Sub might be worthwhile. (earlgray)

Conclusion

This time, we delivered four presentations ranging from core aspects of the Go language to practical techniques. There were also presentations about GopherCon 2024, which were very educational for the organizing members as they learned about the latest developments in Go.

Thank you very much to those who watched live or recording!

Please look forward to the next event! If you want to receive event announcements, please become a member of our connpass group!

Fine-tuned SigLIP Image Embeddings for Similar Looks Recommendation in a Japanese C2C Marketplace

Fri, 08 Nov 2024 14:33:28 GMT

Hello, we are Yuki and Sho, machine learning engineers on the AI/LLM team at Mercari.

In this tech blog, we dive into how we fine-tuned a large-scale Vision Language model on Mercari’s product catalog to create foundational image embeddings for AI teams across the company.

By using the embeddings obtained from the model created this time, we conducted an A/B test in the "Visually Similar Items" section on the product detail page.

Originally, the "Visually Similar Items" section, internally known as "Similar Looks," utilized a 128-dimensional PCA-compressed embedding derived from a non-fine-tuned MobileNet model.

We conducted an A/B test on the "Similar Looks" feature, using image embeddings from our fine-tuned SigLIP model’s Image Encoder in the treatment group. The results demonstrated significant improvements in key performance indicators:

1.5x increase in tap rate
+14% increase in Purchase Count via Item Detail Page

After confirming the positive results of the A/B test, we have released the fine-tuned SigLIP Similar Looks variant to 100% of the users. In this article, we will discuss about the defails of the project including the fine-tuning process, offline evaluation, and the end-to-end deployment infrastructure.

Fine-tuning of the SigLIP model using product data

Image Embedding

Image embedding is a core technique that expresses features such as the objects appearing in an image, their colors, and types as numerical vectors. In recent years, it has been used in various real-world application scenarios like recommendation and search. Within Mercari, its importance is increasing daily. Image embeddings are used in various contexts such as similar product recommendations, product searches, and fraudulent listing detection.

Recently, the AI/LLM team at Mercari worked on improving product image embedding using a large-scale Vision Language Model: SigLIP.

SigLIP

In recent years, models that have been pre-trained using contrastive learning with large-scale and noisy image-text pairs datasets, such as CLIP [3] and ALIGN [4], are known for achieving high performance in zero-shot classification and retrieval tasks.

The SigLIP model was introduced in a paper presented at ICCV 2023. This Vision Language Model employs a novel approach to pre-training by replacing the conventional Softmax loss function used in CLIP with a Sigmoid loss function. Despite the simplicity of this modification, which solely involves altering the loss calculation method, the authors report significant performance improvements on standard benchmarks, including image classification tasks using ImageNet [6].

Let’s examine the implementation of the loss function that was developed for fine-tuning the model using Mercari’s internal dataset, which will be discussed in more detail later.

def sigmoid_loss(
    image_embeds: torch.Tensor,
    text_embeds: torch.Tensor,
    temperature: torch.Tensor,
    bias: torch.Tensor,
    device: torch.device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
):
    logits = image_embeds @ text_embeds.T * temperature + bias
    num_logits = logits.shape[1]
    batch_size = image_embeds.shape[0]
    labels = -torch.ones(
        (num_logits, num_logits), device=device, dtype=image_embeds.dtype
    )
    labels = 2 * torch.eye(num_logits, device=device, dtype=image_embeds.dtype) + labels
    loss = -F.logsigmoid(labels * logits).sum() / batch_size

    return loss

We utilized google/siglip-base-patch16-256-multilingual as a base model. This model has been trained on a multilingual WebLI dataset [5], making it particularly suitable for our application as it supports Japanese, which is the primary language used in Mercari’s service.

Fine-tuning Using In-house Data

In this section, we introduce a detailed setting of fine-tuning experiments of SigLIP using real-world service data. We conducted fine-tuning of the SigLIP model using approximately one million randomly sampled Mercari product listings (text-image pairs) from items listed. The input data for SigLIP consisted of product titles (text) and product images (image), both of which were created by sellers on the Mercari platform.

The training code was implemented using PyTorch and the Transformers library. Due to the large scale of our dataset, we leveraged WebDataset to optimize the data loading process, ensuring efficient handling of the substantial amount of training data.

Model training was conducted on a single L4 GPU. We utilized Vertex AI Custom Training to construct a robust training pipeline. For experiment monitoring, we employed Weights & Biases (wandb), taking advantage of Mercari’s enterprise contract with the platform. This setup allowed for comprehensive tracking and analysis of the training process, facilitating iterative improvements and model optimization.

The combination of these technologies and platforms—PyTorch, Transformers, WebDataset, Vertex AI, and wandb—provided a scalable and efficient framework for fine-tuning the SigLIP model on our proprietary e-commerce dataset, while maintaining close oversight of the training progress and performance metrics.

Offline Evaluation

Prior to conducting A/B testing, we performed an offline evaluation using user interaction logs from the existing "visually similar products" feature. This evaluation utilized approximately 10,000 session data points.

Here is a specific example of an action log. The query_item_id holds the ID of the product displayed on the product detail page as the query image, similar_item_id contains the ID of the product displayed in the "Similar Looks" section, and clicked is a flag indicating whether the product was viewed or not.

session_id      | query_item_id  | similar_item_id | clicked |
----------------|----------------|-----------------|---------|
0003e191…       | m826773…       | m634631…        | 0       |
0003e191…       | m826773…       | m659824…        | 1       |
0003e191…       | m826773…       | m742172…        | 1       |
0003e191…       | m826773…       | m839148…        | 0       |
0003e191…       | m826773…       | m758586…        | 0       |
0003e191…       | m826773…       | m808515…        | 1       |
...

We formulated the evaluation as an image retrieval task, treating user clicks as positive examples. The performance was assessed using nDCG@k and precision@k as evaluation metrics. This approach allowed us to quantitatively measure the model’s ability to rank relevant products in a manner consistent with user preferences.

We conducted our evaluation using two baseline methods for comparison: random recommendation and image retrieval based on MobileNet, which is currently employed in the existing Similar Looks feature.

The following were our results:

Method	nDCG@5	Precision@1	Precision@3
Random	0.525	0.256	0.501
MobileNet	0.607	0.356	0.601
SigLIP + PCA	0.647	0.406	0.658
SigLIP	0.662	0.412	0.660

Evaluation results show that image retrieval using embeddings from the fine-tuned SigLIP Image Encoder consistently outperformed MobileNet-based image search, even when SigLIP embeddings were compressed from 768 to 128 dimensions using PCA. This demonstrates the superior performance of our fine-tuned SigLIP model for product similarity tasks.

In addition to quantitative evaluation, we also conducted qualitative evaluation through visual inspection. We created a vector store using FAISS, containing embeddings of approximately 100,000 product images. We then performed image searches for multiple products and compiled the results in a spreadsheet, as shown below, for visual inspection.

These results conclusively demonstrate that the Similar Looks Recommendation system, powered by the SigLIP Image Encoder fine-tuned on product data, outperforms the existing model both quantitatively and qualitatively. So, we decided to proceed with A/B test using the created model. In the following chapters, we will present the system design for deploying this model to production.

Deployment Architecture

End-to-End Architecture

Before diving into individual components, here’s a high-level view of our architecture:

In the diagram above, you can see how data flows from the marketplace platform to our model services and how embeddings are stored and retrieved efficiently. While this is an initial version, this modular design ensures scalability and flexibility as we evolve the system.

Google Container Registry

Our model deployments are managed through Google Container Registry (GCR), where Docker images of our microservices are stored. These images are continuously built and pushed to GCR from our GitHub repository via a CI/CD pipeline with Google Cloud Build.

By leveraging GCR, we ensure that our deployments in Google Kubernetes Engine (GKE) are always based on the latest versions of the code, offering seamless updates to the services that run in production.

Google Pub/Sub

To handle real-time data streams, we rely on Google Pub/Sub. New listings created on our marketplace are published as messages to specific topics, such as topics for new listings. The relevant microservices subscribe to these topics, enabling the system to react dynamically to new product listings.

Whenever a seller uploads a new product image, a message is sent to Pub/Sub. This triggers our Embeddings Worker, which processes the image from the new listing and updates the vector database with new embeddings. This asynchronous system allows us to scale effectively with the volume of marketplace activity.

Google Kubernetes Engine

The heart of our deployment lies within Google Kubernetes Engine (GKE). This platform hosts several key services in our architecture:

Embeddings Worker

The Embeddings Worker is a critical service that listens to the new listings topic in Pub/Sub. For each new listing, the worker:

Fetches the corresponding image
Converts it into a fixed-length vector embedding using our fine-tuned SigLIP model
Runs Principal Component Analysis (PCA) to reduce the dimensions for improved latency on the similarity search and cost savings for storage (768 dim → 128 dim)
Stores the embedding in Vertex AI Vector Search

This process enables us to perform image similarity searches efficiently. Each embedding represents the visual content of the image, making it easy to compare and find visually similar listings across the platform.

Index Cleanup Cron Job

As the marketplace is highly dynamic, with new listings being added and old listings getting sold or removed, we needed a way to keep our embeddings up-to-date. For this, we implemented an Index Cleanup Cronjob. This cron job runs periodically to remove embeddings corresponding to outdated and sold listings from Vertex AI Vector Search.

While this batch cleanup process works well for now, we are exploring live updates for embedding management to improve efficiency further.

Similar Looks Microservice & Caching

The Similar Looks Microservice is the core of our image similarity feature. It takes a listing ID as input, retrieves the corresponding image embedding from Vertex AI Vector Search, and performs a nearest-neighbor search to find similar items in the marketplace.

To reduce latency, we’ve implemented caching mechanisms in this microservice as well. This ensures a smooth user experience by delivering quick responses when users browse for similar products.

Vertex AI Vector Search

For storing and retrieving embeddings, we use Vertex AI Vector Search, a scalable vector database that allows us to efficiently search for similar embeddings. Each product image in the marketplace is mapped to a vector, which is then indexed by listing ID in Vertex AI.

The nearest-neighbor search algorithms built into Vertex AI enable fast retrieval of visually similar listings, even with a large amount of embeddings in the database.

Model Optimization with TensorRT

To optimize the performance of our fine-tuned SigLIP model and handle our high amount of listings created per second, we converted the model from PyTorch to TensorRT, NVIDIA’s high-performance deep learning inference library. The conversion resulted in a \~5x speedup in inference times.

TensorRT

TensorRT optimizes deep learning models by performing precision calibration, layer fusion, kernel auto-tuning, and dynamic tensor memory allocation. Specifically, TensorRT converts the operations in the neural network into optimized sequences of matrix operations that can run efficiently on NVIDIA GPUs.

For our marketplace, this improvement was critical. With a massive amount of product listings being created per second, reducing inference time from hundreds of milliseconds to mere fractions enabled us to make sure that all new listings have their images almost instantly embedded into vectors to be ready in the Vertex AI Vector Search index for the Similar Looks component to use.

Next Steps

While our current deployment architecture is stable and scalable, we are constantly looking for ways to improve. Here are some of the next steps we are working on:

Live Updates of Embeddings

Currently, the Index Cleanup Cronjob is responsible for removing outdated embeddings from Vertex AI Vector Search. However, we plan to move to a more real-time solution where embeddings are updated as soon as a listing is removed or sold. This will eliminate the need for periodic cleanups and ensure that our index is always up-to-date.

Triton Inference Server

We are also exploring the use of Triton Inference Server to handle model inference more efficiently. Triton allows for the deployment of multiple models across different frameworks (e.g., TensorRT, PyTorch, TensorFlow) in a single environment. By shifting inference from the Embeddings Worker to Triton, we can decouple the model execution from the worker logic and gain greater flexibility in scaling and optimizing inference performance.

New Features Using the Fine-Tuned SigLIP Model

Lastly, we are working on new features that will leverage our fine-tuned SigLIP model. Stay tuned for updates on how we plan to enhance the user experience with advanced image search capabilities, potentially including multimodal search, where users can combine text and image queries to find exactly what they are looking for, as well as apply the embeddings to a lot of different Mercari features and processes.

Conclusion

In this project, we fine-tuned the Vision-Language Model SigLIP using Mercari’s proprietary product data to build a high-performance Image Embedding Model, improving the "Visually Similar Items" feature.

In offline evaluations, the fine-tuned SigLIP demonstrated superior performance in recommending "Visually Similar Items" compared to existing models. Consequently, when we conducted an A/B test, we observed significant improvements in business KPIs.

We hope that the content of this blog will be helpful to those interested in fine-tuning Vision Language Models, evaluation, and deploying deep learning models to real-world services.

Mercari is hiring Software Engineers who want to make impactful product improvements using Machine Learning and other technologies. If you’re interested, please don’t hesitate to apply!

References

[1] Sigmoid Loss for Language Image Pre-Training, 2023
[2] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, 2017
[3] Learning Transferable Visual Models From Natural Language Supervision, 2021
[4] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, 2021
[5] PaLI: A Jointly-Scaled Multilingual Language-Image Model, 2022
[6] ImageNet: A Large-Scale Hierarchical Image Database, 2009

Fine-Tuning an LLM to Extract Dynamically Specified Attributes

Fri, 13 Sep 2024 12:07:47 GMT

Hello, I am @andre, a machine learning engineer on the AI/LLM team at Mercari.

In a previous article, we discussed how our team utilized commercial LLM APIs to build an initial feature to support our customers and improve the platform’s selling experience.

This article will describe one of our past experiments in fine-tuning a 2-billion parameter large language model (LLM) using QLoRA, to extract dynamically specified attributes from user-generated content, and compared the performance with GPT-3.5 turbo—a much larger model. Results show that the fine-tuned model outperforms the bigger model in terms of extraction quality while being significantly smaller in size and less costly. We hope this article will provide valuable insights into what it takes to fine-tune an LLM effectively.

Background

In a Japanese customer-to-customer (C2C) marketplace, specific details could impact the quality of a listing description. However, understanding the precise details in a user-generated listing description can be tricky. This is due to several challenges, including:

Wide variety of user-generated content: Each seller describes their listings differently.
Category specificity: What’s essential varies from one category to another.
Time sensitivity: User-generated content continuously evolves.

By accurately extracting existing key attributes from listing descriptions, we can gain a deeper understanding of the contents written by our customers—specifically, in this case, the sellers. Figure 1 below illustrates an example of a listing description and the extracted values. For the purpose of this article, the illustration shows an example of a listing written in English; however, most listings within Mercari are written in Japanese. Such insight can also help us guide our customers to enhance their listings, making them more appealing and effective.

Figure 1. Illustration of the extracted attributes from a sample listing description

Why not just use light-weight, conventional, non-LLM models?

Dynamic and varied attributes: The way attributes are described can change frequently, leading to high maintenance requirements and the need for continuous model re-training. Having a model that could handle dynamically specified attributes could go a long way.
Generalization capability: Large language models (LLMs) have the potential to generalize far better than conventional ML models with much less training data, even for handling out-of-distribution data.
Multi-linguality: Most listings in Mercari are written in Japanese, however, with the huge variety of goods being exchanged, there are also listings written in other languages, such as English and Chinese. The multilingual capability of recent LLMs are expected to be able to handle such varieties better than conventional ML models.

On the other hand, why not just use existing commercial LLM APIs?

Cost of commercial APIs: Though commercial LLM APIs are becoming more affordable, at the time this article is written, the sheer number of requests in a production environment would still make them prohibitively expensive.
Control over hallucinations: It’s more difficult to manage and minimize hallucinations purely through prompt engineering with commercial APIs.

Given these considerations, we decided to experiment with fine-tuning our own model. For this experiment, we used a single A100 GPU with an 80 GB memory VM instance (a2-ultragpu-1g) from GCP to fine-tune a large language model using QLoRA. Our short-term goal was to see if we can build a model that could achieve similar or even better performance than GPT-3.5 Turbo despite being significantly smaller and cheaper to run in production.

Dataset and Base Model

To tackle our task, we first defined the input and output requirements for the model:

Input: A text description of the listing and a list of attribute keys to extract. For example:
- Listing description: A Mercari T-shirt size M, blue. Used only once and kept in a clean wardrobe after.
- Attribute keys: size, color, original retail price
Output: The extracted attributes and their values. For example:
- Size: M
- Color: Blue
- Original retail price: NONE

To build our dataset, we gathered historical descriptions along with their attributes. Since attribute keys can vary across item categories, we started by focusing on the 20 categories with the highest listings on our platform.

We structured the data into inputs and outputs and integrated these pairs with specific prompts, which were then used to fine-tune the LLMs. We experimented with various prompts written in English and Japanese; however, the prompt generally contains the following.

An initial prompt sentence, telling the model that it will receive an instruction below and instructing it to respond accordingly.
The instruction, mentioning that it will be given a description text in the context of an online marketplace listing, and instructing the model to extract a list of attribute keys from the input text. It also tells the model to respond following a specific format.
The input text, containing the listing description text from which we want to extract attributes.
The output text, containing the response text with the attribute keys and the extracted values.

Below is an example of the prompt templates we experimented with, written in Japanese:

以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。
リクエストを適切に完了するための回答を記述してください。

### 指示:
次の文章はオンラインマーケットプレイスに投稿されているリスティングの情報です。
その文章から{attr_names}の情報を探し出してください。
妥当な情報が存在したら「{attr_name}: <内容>」で応答してください。逆に存在しない場合はかならず「{attr_name}: NONE」で応答してください。

### 入力（文章）:
{input}

### 応答:
{output}

Once the dataset was ready, our next step was identifying potential LLMs for fine-tuning. The Nejumi Leaderboard for Japanese LMs, curated by the Weights and Biases Japan team, was one of our primary resources. It comprehensively evaluates various large language models’ capabilities in handling Japanese text. After testing and experimenting with several models, we decided to move forward with the gemma-2b-it model provided by the team at Google (paper, HF).

Parameter efficient fine-tuning with QLoRA

To embark on our fine-tuning journey, we used QLoRA—a cutting-edge approach known for its efficient fine-tuning. As cited from the original paper, QLoRA significantly reduces memory usage, allowing one to fine-tune a 65B parameter model on a single 48GB GPU while preserving the full 16-bit fine-tuning task performance. The image below illustrates how QLoRA compares to full fine-tuning and LoRA methods.

Figure 2. Illustration of how fine-tuning with QLoRA works under the hood (adapted from the original figure on QLoRA: Efficient Finetuning of Quantized LLMs)

Now, let’s dive into the fine-tuning process!

Initially, we load the pre-processed dataset previously stored as W&B artifacts into memory.

...
with wandb.init(entity=ENTITY_NAME, project=PROJECT_NAME, job_type=JOB_TYPE_NAME, tags=["hf_sft"]):
    artifact = wandb.use_artifact(ENTITY_NAME+'/'+PROJECT_NAME+'/train_test_split:latest', type='dataset')
    artifact_dir = artifact.download()

loaded_dataset = load_dataset("json", data_dir=artifact_dir)
train_data = loaded_dataset["train"]
eval_data  = loaded_dataset["test"]
...

Then, we define the LoRA configurations (hyperparameters) and target modules. One example of the modules and configurations that we experimented with is as follows:

...
target_modules = ['q_proj','k_proj','v_proj','o_proj','gate_proj','down_proj','up_proj','lm_head']

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    target_modules = target_modules,
    task_type="CAUSAL_LM",
)
...

And then, the fine-tuning hyperparameters and quantization configurations. Following is an example of the configurations that we experimented with:

...
training_args = TrainingArguments(
    output_dir=base_dir,
    report_to="wandb",
    save_strategy="epoch",
    evaluation_strategy="epoch",
    num_train_epochs = 1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    optim='adamw_torch',
    learning_rate=2e-4,
    fp16=True,
    max_grad_norm=0.3,
    warmup_ratio=0.1,
    group_by_length=True,
    lr_scheduler_type="linear",
)

nf4_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_use_double_quant=True,
  bnb_4bit_compute_dtype=torch.bfloat16
)
...

Once the above are set up, we then load the base model and tokenizer from HuggingFace:

...
model_path = "google/gemma-2b-it"
tokenizer = AutoTokenizer.from_pretrained(model_path, add_eos_token=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map='auto', quantization_config=nf4_config,
)
...

We then use the SFTTrainer from HuggingFace to begin fine-tuning:

...
trainer = SFTTrainer(
    model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    packing=True,
    max_seq_length=1024,
    args=training_args,
    formatting_func=create_prompt,
)
# Upcast layer norms to float 32 for stability
for name, module in trainer.model.named_modules():
  if "norm" in name:
    module = module.to(torch.float32)

run = wandb.init(entity=ENTITY_NAME, project=PROJECT_NAME, job_type="start_finetuning", config=config)
st = time.time()
trainer.train()
elapsed = time.time() - st
run.log({"elapsed_time (seconds)": elapsed})
run.finish()
...

Finally, we merge and save the fine-tuned model:

...
new_model = NEW_MODEL_PATH_AND_NAME
trainer.model.save_pretrained(new_model)
trainer.tokenizer.save_pretrained(new_model)

base_model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
)
merged_model= PeftModel.from_pretrained(base_model, new_model)
merged_model= merged_model.merge_and_unload()

merged_model.save_pretrained(new_model+"-merged",safe_serialization=True)
tokenizer.save_pretrained(new_model+"-merged")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
...

Post-training Quantization and Model Evaluation

Post-training quantization aims to see if we can further shrink the model size while maintaining satisfactory performance. We used the llama.cpp library—an open-source tool that enables post-training model quantization and faster inference using LLMs in C/C++.

Here’s an overview of the steps we followed using llama.cpp for model conversion and quantization. Note that some steps might be outdated by the time of publication, so we recommend referring to the llama.cpp repository for the latest information:

Clone the Repository: Clone the llama.cpp GitHub repository and run the build commands using the appropriate settings. Detailed instructions can be found here.
- Note: Since support for Gemma models was added around the end of February 2024, ensure you use the correct version of llama.cpp.
Convert the Model: Convert the fine-tuned model, previously stored in the HuggingFace format, to a format compatible with llama.cpp.
Select Quantization Method: Choose the quantization method and start the quantization process. The 4-bit precision method (q4_k_m) worked well for our use case.
Convert and Quantize: the resulting model is stored in the GGUF format.

After the post-training quantization finishes, we evaluated the model in GGUF format and compared its performance. As of our experiment, GPT-4o (including the mini model) was not available. Therefore, considering its cost and latency advantages, we chose GPT-3.5 turbo (specifically, gpt-3.5-turbo-0125) as our baseline model for performance comparison.

Some key metrics for the evaluation:

BLEU Score: This score provided insights into the quality of extracted attribute values compared to the actual values.
Model Size and Latency: We also checked the resulting model size and latency to assess cost-efficiency and readiness for production use.

Here are some key findings from our quick experiment:

The final 4-bit precision GGUF model (q4_k_m) is a QLoRA fine-tuned version of the gemma-2b-it model.
The model is approximately 95% smaller than the gemma-2b-it base model downloaded from HuggingFace.
The model achieved a BLEU score slightly more than five percentage points higher than gpt-3.5-turbo-0125.
Additionally, an initial rough estimate at the time of the experiment showed that using the fine-tuned model could reduce the cost by more than 14 times compared to using gpt-3.5-turbo-0125. However, given the rapidly changing pricing structures of commercial models, this figure should be taken with a grain of salt.

In summary, the final model is significantly smaller—approximately 95%—than the original base model from HuggingFace and achieves a BLEU score higher than gpt-3.5-turbo-0125.

Conclusion

This experiment demonstrates the practicality of fine-tuning our LLM for attribute value extraction from user-generated content as an effective alternative to commercial LLM APIs. By utilizing QLoRA, we managed to fine-tune the gemma-2b-it model efficiently, reducing its size by around 95% compared to the original base model. Despite this significant size reduction, our fine-tuned model still outperformed gpt-3.5-turbo-0125 by achieving a higher BLEU score, thus validating the efficacy of our approach in both performance and resource optimization.

Besides the improvements in performance and cost savings, our hands-on approach provided better control over the model’s behavior, helping to mitigate issues like hallucinations more effectively than prompt engineering alone. We hope this article offers valuable insights and practical guidance for those looking to fine-tune their models and transition away from expensive and less controllable commercial APIs. By leveraging advancements in large language models and innovative techniques like QLoRA, there are significant opportunities for future development and optimization.

Mapping the Attack Surface from the Inside

Mon, 22 Jul 2024 11:08:15 GMT

Abstract

If a company wants to protect its attack surface, it first needs to know it, yet in many companies, there is no clear picture of what services are exposed to the internet. We have been working on a system to create a map of the company’s attack surface. There are many explanations of this process from the perspective of the attacker, but it turned out to be a very different process from the inside.

At Mercari, we currently allow a lot of flexibility to developers on what they deploy and how they deploy it, which means there is a large variety of places we have to check if we want to create a complete inventory. We attempted to create a system that requires minimal maintenance and contribution from individual developers while still granting good oversight of our infrastructure, weak points, and services we can deprecate. In the process, we gained a better understanding of our infrastructure and learned about the pitfalls of relying on IaC. We have also learned to embrace flexibility in designing a system that is mapping the unknown. When you plan to handle things you are just now discovering exist, your first plan will likely not be correct.

Security Philosophy

Before making a plan, I think explaining the security philosophy informing our design decisions is useful. We tend to prefer solutions that put the least burden on developers since the more efficient their work is, the more they can deliver on the product side. At the same time, we have to make solutions that scale to the size of a fairly large company.

Kelly Shortridge wrote a blog post back in 2020 about the problems of over-doing and under-doing security that was very impactful for me. The problem with creating an overly strict security environment is that it suffocates the organization. Developers are bogged down by waiting on security reviews and prevented from using the latest and greatest technology.

The Managerial Security Mindset

Creating a rigid system is a really easy mistake for a security professional. If the job is to make everything secure, one can hardly be blamed for wanting control over everything. It is a managerial mindset in which the security team tries to guide secure development through restrictions, reviews, and fixed rules of what can and cannot be done in the company. The problem with this attitude is not only that no company has enough security engineers to manage absolutely everything but also its complete antagonism towards innovation.

Companies need to create things to make a profit, and if they want to stay ahead of the competition, they need to use the latest technology to create those things. In the managerial security mindset, everything outside of the mold is scary, full of unknown risks that will definitely destroy the company. In reality, developers experimenting with new solutions and project managers experimenting with new features are the things that propel the company forward. While most new technologies and ideas might not be great, if experimenting itself is made to be a burden, the company will stagnate, calcify, and will eventually go bankrupt by more innovative corporations delivering a better product faster, even if not quite as securely.

The Importance of Developer Attitude

It is also worth keeping in mind that if security processes become annoying and tiresome, their efficiency falls off a cliff. Most developers are interested in security and will willingly contribute to improving it, provided they aren’t hampered by excessive procedural hurdles. On the other hand, once the amount of security procedures becomes a hindrance, it will create an adversarial relationship between the security team and developers.

With these considerations in mind, our approach focuses on empowering developers by providing them with intuitive tools and clear security information. Instead of constraining their technological choices, we expand our visibility to understand and secure these technologies collaboratively.

Finding the Sweet Spot

Naturally, the reality is somewhere in the middle. Sometimes, restrictions are necessary, and some security burden has to be placed on the developers. I think an ideal security posture is not just halfway between complete rigidity and complete chaos. The sweet spot is constantly moving depending on market trends, technical innovations, and ultimately, what the business is trying to achieve.

Initial Plan

The original PoC for this project aimed at detecting new domains added to one of our sub-companies so they could be added to Burp Enterprise for periodic scanning. To achieve this, we simply have to parse the IaC repositories that contain the domains, and present the new ones to the team every week. Once a team member makes a decision, we can use the Burp Enterprise API to schedule scanning for the domain.

Implementing the Burp Enterprise API

At the time of creation, there was not much documentation on how to use PortSwigger’s Burp Enterprise API. There is a REST and a GraphQL API with different capabilities. The REST API is lacking a lot of features we need, since it is just a slightly changed version of the Burp Professional API. The GraphQL api provides most of the functionality we need, but there is no way to pin the API version and it is still under development, so we are risking features breaking on every update. Still, it is either the GraphQL API, or Selenium, so GraphQL it is. With a GraphQL API, we are expected to hand-craft the specific requests we want to use. Given the vague documentation, this seemed fairly time consuming.

Looking for an easier option, we’ve stumbled upon genqlient from Khan Academy. For the correctly formatted GraphQL schema, genqlient can create a go library accessing all the queries and mutations of that schema. It is not perfect, but after a bit of tweaking, it works fairly well. PortSwigger does not publish its schema, but the default installation allows GraphQL Introspection. During penetration testing, an attacker might use this to better understand the capabilities of the API. In this case we used it for the same reason, but we intend to legitimately use the API.

To create a complete introspection query, we used gqlfetch because it immediately formats the results into a standard format that can easily be converted to SDL. After you have the resulting SDL file, you can generate individual query and mutation files with gqlg

gqlg --schemaFilePath schema.graphql --destDirPath ./gqlg --ext graphql

The resulting ./gqlg folder will have a list of queries and mutations, from which you can select the ones you want to use. We simply copied the useful ones into the ./used_query_schemas/ folder and capitalized their name to make the corresponding Golang functions exported. Some of the files might be partially incorrect, for those cases you’ll have to rename some things or address errors as they arise.

go run github.com/Khan/genqlient

This will generate the Go library. If you tweaked the gqlg files correctly, this library should compile and export functions to interact with the API. You’ll also have to implement an authentication RoundTrip to add the “Authorization” header with the Burp API key.

After getting over that hurdle we tried using this solution for the first time.

We used a Slack bot to create a simple, interactive Slack message where knowledgeable team members could decide whether a domain should be scanned.

Initial Learnings

When we started to use this slack bot, a few things became clear. There are a lot of websites and a lot of new subdomains registered every week, making a decision on them still requires manual labor. It is often not obvious what a domain is used for, their names range from legible words to 12 character random strings. The sites hosted range from test sites to pages that simply respond with 404. Most of the websites are hosted by us, but some of them are handled by third parties that we should not scan. Most importantly, there are a lot more websites owned by the company than what we parsed so far. They can be found in a variety of different IaC repositories responsible for different departments, or CDN configurations. Some domains are simply defined directly in the cloud without any IaC and some services do not have a domain at all.

The tragedy of IaC

I mentioned that the approach of parsing IaC did not quite work out. This was not because we were unable to parse the fairly large number and variety of IaC repositories that all define different services. It was ultimately because IaC is simply inaccurate.

https://en.wikipedia.org/wiki/Allegory_of_the_cave#/media/File:An_Illustration_of_The_Allegory_of_the_Cave,_from_Plato%E2%80%99s_Republic.jpg

We spend a lot of time writing IaC code to define all kinds of resources, but half the time it does not work and sometimes it cannot work. For example, there are some features in GCP that the terraform provider simply does not support, or if it does, it is documented so badly that people will sooner give up and set it from the gcloud cli or on the web console. Every time that happens, a discrepancy between IaC and reality is created.

That is all to say, IaC is more of an approximation of the infrastructure, and less of a concrete definition. Of course, we do our best to ensure accurate IaC for critical infrastructure, but the things we are most interested in are anything but. We want to see accidentally published services in test environments, long forgotten infrastructure created before the widespread adaptation of IaC and the like.

Going to the Source

To solve the issues of IaC, we decided to switch to directly querying the asset inventory of the various cloud providers. Luckily, GCP, AWS and hopefully Azure (although we haven’t gotten that far yet) have their own inventory of what assets they are housing. This includes not only hosted zones and Route53 configurations, but also things like IP addresses, or ephemeral services such as GCP’s Cloud Run.

These are especially interesting, since they form part of the attack surface without requiring a domain or a dedicated IP address. In GCP there is both an “Asset Inventory” and a “Security Asset Inventory”, in which the security one seems to be easier to query. In AWS, you can use an AWS Config fed by an Aggregator to create a similar inventory. With this approach, we have a more complete picture that is also more accurate. Even if a developer bypasses IaC to create a domain or resource, we will be able to see it. In some cases we also get the user who created the resource, giving us a good idea on who to contact if we find an issue.

Visualization

After we set this collection system up, it quickly became clear that some visualization would make the data more useful. Questions like “which sites are reachable from the internet”, “Are these sites all protected by Identity-Aware Proxy (IAP)?” arose during development, which we could answer at a glance once we made screenshots of every site. We were also able to spot anomalies, like unexpected services being hosted, and domains that pointed to IP addresses that were now in use by other tenants in the cloud.

To do this, we have set up a Google Cloud Run (GCR) service that accepts a list of domains, and spins up chromium to take screenshots of them. Utilizing the automatic scanning of GCR, we batched the domains in a daily GCR job and spun up a few dozen instances to take all the screenshots in about 10 minutes.

We were also able to create connections between domains and IP addresses. This meant that we no longer had to manually review every domain before scanning. If we know that a domain points at an IP owned by our cloud tenant, we can simply add it to Burp Suite and wait for the results to roll in.

Conclusion

When starting the project it was only meant to be a way to automate the mundane process of adding domains to Burp Enterprise. The initial PoC got us closer to that goal, although it still proved to be too burdensome to use. To fix that, we had to add some functionality and change some existing features. We then had to move away from relying on IaC and pivot to using cloud inventories. Then we decided to be more ambitious and change the system into a complete attack surface inventory.

During this project we have learned a lot about our infrastructure. Knowledge about the attack surface is held in as many parts as the people who have created it. Consolidating that information into one place gives us a great ability to detect weak points and anomalies. Perhaps the weakest points of our attack surface were the ones that we knew the least about. Sites created years ago now lay abandoned, as their creators moved on to new projects. The older a system is, the less likely it is to be using recent solutions, like IaC or even the Cloud, and the more likely it is to not be maintained. Long forgotten, and with little detectable evidence of their existence, these systems still churn away, waiting to serve users and attackers alike. The things we need to see the most are the best hidden.

With every iteration we not only added new features, but also changed and undone some things we already spent time working on. This may seem like a waste of time, but in practice, almost every process works this way. When the process is started, the way to get to the final goal is often not known. We start on a path, and periodically reassess to see if we are getting closer. As we get closer to our goal, we might realize we were slightly off-course and need to correct, or we might even realize that our goal was not as useful as a different goal we are also approaching. We should be ready to adapt during the project to deliver the best thing we can, even if it is different from our initial goal. When I feel stuck on a project, I find it helpful to simply start doing anything, and oftentimes that work will produce information that helps me find a good direction for the next step.

Mercari Ranked #1 in Technology Branding Ranking for three years in a row!

Tue, 16 Jul 2024 18:21:21 GMT

Hello, this is yasu_shiwaku from the Engineering Office.

On July 16th 2024, Mercari was awarded first place in "Technology Branding” at the Developer eXperience AWARD 2024 conducted by the Japan CTO Association, for the third consecutive year. The press release annoucement by the Japan CTO Association is available here.

The Award ceremony was held in-person in Tokyo following the previous year’s event. Shunya Kimura, CTO Marketplace of Mercari, attended the event to receive the plaquette (Kimura is presenting as a panelist on July 17th’s panel discussion at the same event)

We are pleased to receive high evaluations from many people in the Tech industry in Japan for three years in a row. This is thanks to our engineers who contribute to the technical output on a daily basis, in a wide variety of ways such as blogs, presentations and attending events, both internally and externally.

Mercari Group is fostering a culture in which engineers proactively communicate and give back their experience and knowledge to the technology community, to aid in empowering the industry as well as helping it grow.

We also contribute to the open source community by supporting conferences, project sponsoring and other various supporting activities (see here for Mercari’s standpoint on open source. The softwares open to the public is here)

Under the mission to “Circulate all forms of value to unleash the potential in all people,” the members of Mercari Group will proactively continue to disseminate information to contribute to the development community, in order to circulate the values which our Engineering Organization can provide.

List of Engineering contents platform

Mercari Engineering Website (this portal site)
X account (English, Japanese)
Events related
- Connpass
- Meetup
YouTube Channels
- Mercari Gears
- Mercari devjp

If you are interested in what kind of developer experience and culture you can have at Mercari Group, please take a look at our career site!
Software Engineer/Engineering Manager