Enabling internationalization in our web Turbo monorepo

Hello, Gary here again! If you haven’t had a chance already, please be sure to check out my earlier blog post on the motivation for developing a new global service in addition to the other articles by my team in our series here.

Background – One repo per service

Here at Mercari, we have a solid platform for provisioning infrastructure using terraform, configuring kubernetes, setting up CI/CD pipelines, et cetera, for our services. Specifically for web application development, we also have a plethora of npm packages that help with the initial setup and configuration for what we call our “golden path” for web application development. A lot of the complexity for creating a new web application is abstracted away making it easier for teams to focus more on the meaty part of the process, writing the business logic and UI.

In recent times especially, we’ve seen an explosion of new web applications. As a company we are striving for agility and the ability to quickly provision new services to test out hypotheses about new potential businesses. We’ve found that beyond foundational core application concerns, there is also a considerable amount of logic and UI that is shared across these applications. Utilities for handling theming, URL manipulation, experiment configuration, logging and so on.

We discovered some inefficiencies in our process as due to siloing applications in their own git repositories. Particularly:

• Sharing of common code is difficult. Sure, we can create npm packages, but doing so involves considerable effort and creates a barrier for collaboration.
• Configuration and maintenance of GitHub workflows is time consuming and requires expertise.

Consequently for the new global service, we decided rather than adding a new silo to the farm, we would instead challenge ourselves to move to a monorepository in order to solve these two pain points. Given we already had the existing Japan Marketplace application and were building a new global application with similar product specifications, we decided that converting that existing repository into a monorepo was the cleanest path forward to enable code sharing in the future.

The move to modularization

To ensure a clear separation of concerns and allow for long-term scalability we opted for a modular monorepository with individual npm packages separating applications and packages.

We opted for pnpm given its speed, workspace protocol for managing internal dependencies, and its ability to create catalogs for managing shared versions across multiple packages (e.g. pinning the whole monorepo to a single React version).

For the build system and script management we decided to use Turborepo, again for its speed and ability to configure complex build pipelines.

Similar to the architecture our backend team already adopted for their services, we wanted to define modular hierarchy and relationships to encourage consistent practices for module development. We initially started off with 5 levels, but soon scrapped one (the page layer) opting to reduce the complexity a little.

We define each layer as such:

• App: The Next.js application responsible for setting up global configuration such as instrumentation (logging) for the application, providing the root layout along with global context providers, and finally for connecting routes to other modules as pages
• Page: Compositions of Feature/Domain/Core modules that can be imported into one or more apps (but since reuse potential for Page modules seemed very low we decided they didn’t offer much benefit)
• Feature: The most common type of module, containing business logic and UI code for a specific product feature. Not necessarily tied to a single page
• Domain: Anything that addresses application specific concerns that needs to be shared across multiple features.
• Core: Essential libraries that contain non-business logic and/or non-product-specific UI that is shared across multiple domains and/or features.

Our monorepo looks something like this:

app > …
feature > …
domain > …
core > …
package.json
pnpm-workspace.yaml

Internationalization

With the modular monorepo architecture set up, we were considerably more enabled to reuse code across multiple applications. For utilities and logic, this is relatively straightforward. However for UI, it’s a little more complicated, especially considering that different applications have different internationalization requirements.

We are using i18next as the base library as it gives us a lot of functionality out of the box including string interpolation, pluralization, formatting etc. But it unfortunately does not support modularization, so we had to engineer a solution on top of i18next to cleanly implement internationalization in our modules.

To give an example, let’s say we have a feature module for a buy now button that is used both on our new global service and existing Japan marketplace.

The simplest option is just to explicitly import the required translations per applications but this doesn’t scale well:

// app/global/src/translations.ts
import buyNowEn from '@feature/buy-now/translations/en.json'
import otherFeatueEn from '@feature/other-feature/translations/en.json'
import anotherFeatureEn from '@feature/another-feature/translations/en.json'
...
import buyNowZh from '@feature/buy-now/translations/en.json'
...

const languages = ['en', 'zh']

export function loadTranslations() {
  return {
    en: {
      ...buyNowEn,
      ...otherFeatureEn,
      ...anotherFeatureEn,
      ...
    },
    zh: {
      ...buyNowZh,
      ...
    }
  }
}

For the global web service, we already have over ten modules and plan to rollout to 50 regions in the next couple of years. With the addition of more features in the near future this would mean over 500+ lines of configuration…

This N (No. modules) x M (No. languages) complexity is not scalable.

We instead decided on a strategy where each module exposes a webpack import context that can be used by each application to fetch only the translations that are required at build time and store these in the application docker image.

For those unaware, webpack (and other bundlers) creates an import context when you use variables within ES6 import statements.

// feature/buy-now/src/i18n.ts
export async function getTranslationsForBuyNow(language: string) {
 return (await import(`./translations/${lang}.json`)).default
}

Inside each application we then have the following code to fetch the required translations for the configured languages.

// app/global/src/translations.ts
import { getTranslationsForBuyNow } from '@feature/buy-now'
import { getTranslationsForOtherFeature } from '@feature/other-feature'
import { getTranslationsForAnotherFeature } from '@feature/another-feature'
...

const languages = ['en', 'zh']

const features = [getTranslationsForBuyNow, getTranslationsForOtherFeature, getTranslationsForAnotherFeature, ...];

export async function loadTranslations() {
 return languages.reduce((prev, lang) => {
  prev[lang] = Object.assign({}, ...await Promise.all(features.map(getTranslations => getTranslations(lang))));
  }, {})
}

This keeps the configuration instead to the order or N + M, much simpler.

So how do we use the translations?

We now have our big object of translations in the app module, but how do we use those inside the features? At first it seems like the only way is a circular dependency but we can get around it using a nifty trick with bundler aliases.

We have a core module that provides i18n support for all our features, including the ability to render strings in the configured language. We have two flavours of the API, one for client side during React client-side component render and the other for server-side rendering in React Server Components

// Client component
import { useTranslation } from '@core/i18n/client';

function MyClientComponent() {
  const { t } = useTranslation();

  return <>{t('page.component.key')}</>;
}

// Server components

import { getTranslations } from '@core/i18n/server';

async function MyServerComponent() {
  const { serverT } = await getTranslations();

  return <>{serverT('page.component.key')}</>;
}

For brevity I will focus on the server side implementation, but the client side is very similar just with the use of a React Context Provider we initialize with the translations in the root layout that are serialized and sent to the browser.

For the server side we can look at a simplified version of getTranslations:

import { getLocaleFromPath } from '@core/url/server';
import { loadTranslations } from '@alias/i18n-config';

import i18n from 'i18next';

export async function getTranslations() {
  const resources = await loadTranslations();

  await i18n.init({
    resources,
  });

  return {
    serverT: i18n.getFixedT((await getLocaleFromPath())),
    i18n,
  };
}

The main part to take note of is import { loadTranslations } from '@alias/i18n-config';.

This may seem like dark magic but it’s actually pretty simple. In our next.config.js file inside our application we create a simple import alias that maps ‘@alias/i18n-config’ to the ‘app/global/src/translations.ts’ file created before.

// app/global/next.config.js

const nextConfig = {
  webpack: (webpackConfig, options) => {
    config.resolve.alias = {
      '@core/i18n/config': path.resolve(webpackConfig.context,"src/translations.ts")
    }
  }
}

module.exports = nextConfig

So the @core/i18n package pulls in the application translations and uses those translations to initialize the i18next instance that we then use to render strings for our UI. Nice!
Moving forward and improved configuration
This pattern has proven to be stable and effective so we are now planning to convert some of our other core modules to be configurable using this same mechanism. This should hopefully reduce the number of domain modules we have that simply wrap core modules with some small application-specific configuration.

For example, currently we need to wrap our core module for experimentation in a domain module with the available feature flags for a specific web application. By converting “@core/experimentation” to also use bundler aliases we will be able to instead just define feature flags in our application and have the core module directly reference those without the need to maintain a separate module. A nice DX and maintainability improvement, enabled by our modular architecture.

Hope you enjoyed the read! Stay tuned for more updates.