WordPress Plugin i18n, Webpack, and Composer

By: Brad Jorsch

A lot of work has been going on in the Jetpack plugin lately. We have UIs built in React, with the JavaScript bundles being created by Webpack. We have Composer packages used for code sharing, increasingly so as we look into creating standalone plugins like Jetpack Backup and Jetpack Search. And we want everything to be translated for people who speak languages other than English.

A few months back we started getting reports that some translations in Jetpack had gone missing. As we looked into it, we eventually found no fewer than six different ways that translation was broken!

  1. JavaScript bundles weren’t being scanned due to bad file naming.
  2. Webpack’s optimizations were breaking the i18n function calls so WordPress.org’s translation infrastructure couldn’t find them.
  3. Lazy-loaded Webpack bundles weren’t lazy-loading translation data.
  4. Shared React component translations didn’t work in plugins other than Jetpack itself.
  5. Bundled Composer packages weren’t being scanned.
  6. Composer package translations didn’t work in plugins other than Jetpack itself.

It took us a few months, but we’ve now fixed them all. This post will describe how we did it.

Background: How plugins get translated

The recommended way to have your plugin translated is to let WordPress.org extract the translatable strings from your plugin, then build language packs for you based on the work of volunteer translators.

In your code, both PHP and JavaScript, you pass translatable strings to functions such as __()_x(), and so on. When that code is uploaded to WordPress.org SVN, it gets scanned for these calls. The strings for each call are collected and passed into the GlotPress installation at translate.wordpress.org, where volunteers translate them into various languages. The translations are later collected into language packs, which can be downloaded and installed into WordPress so people can experience your plugin in their own language.

(Aside: The extraction is part of the WP-CLI tool: wp i18n make-pot. They use the --slug and --ignore-domain options. Generation of JavaScript translation files is done in a manner similar to wp i18n make-json, but skipping any JS files in a src/ directory)

The extracted strings are all associated with a “text domain” matching your plugin’s slug. If the domain parameter passed to __() and so on doesn’t match, your translations won’t be found at runtime.

All this makes some assumptions about your code, some of which turned out not to be true for the way we were doing things in Jetpack.

Problem 1: JavaScript bundle naming

When we dropped support for Internet Explorer 11 in our JavaScript, Babel stopped transpiling modern syntax such as template strings into an ES5 form that IE11 could understand. This then broke when we deployed to WordPress.com, as that environment automatically applies it own minifier to JavaScript and CSS while serving it and their minifier doesn’t understand template strings either. WordPress.com doesn’t apply its minifier if the files are named like “bundle.min.js”, so we renamed our files like that.

But that ran into one of the WordPress.org translation infrastructure’s assumptions: they assume any “bundle.min.js” can be ignored because it will have a corresponding “bundle.js” next to it. 😬

We didn’t want to include several hundred K of non-minified JS in Jetpack though. Jetpack already has an undeserved reputation for being “bloated”, and the extra files wouldn’t help even if the non-minified JS is never used.

Solution: The URL passed to wp_register_script() can include a query part, and WordPress.com’s minifier can also be bypassed by including minify=false in the query part.

Then we took it a step further. The registration of a Webpack bundle usually involves a fair bit of boilerplate since you also need to read the information produced by @wordpress/dependency-extraction-webpack-plugin and often register some CSS too, something like

$relative_to = __FILE__; // Or something.
$assets = require dirname( $relative_to ) . '/build/bundle.asset.php';
    plugin_url( 'build/bundle.js', $relative_to ), // TODO: Add "?minify=false".
wp_set_script_translations( 'handle', 'textdomain' );
    plugin_url( is_rtl() ? 'build/bundle.rtl.css' : 'build/bundle.css', $relative_to ),
    array( /* Some other dependencies? */ ),

So we added a method in our automattic/jetpack-assets Composer package to handle all that in an easier way. And we can have it perform some simple checks, like requiring that a textdomain be given if the dependencies include wp-i18n.

Assets::register_script( 'handle', 'build/bundle.js', __FILE__, array( 'textdomain' => 'domain' ) );

Problem 2: Webpack optimization breaking i18n function calls

The extraction of translatable strings from JavaScript depends on seeing the call to a function or method named __()_x()_n(), or _nx(), with the various parameters being passed as literal strings. These calls may be preceded by a “translator comment” which is also extracted.

In its default configuration, Webpack in production mode is likely to rename these functions to single-character names and to throw away those translator comments. And even if it’s working now, a changed configuration or a new code pattern might break it in the future (as happened to us when we updated to Webpack 5). 😬

Solution, part 1: The first step was to figure out the necessary configuration to preserve the i18n function calls and the translator comments.

  • Set Webpack’s .optimization.concatenateModules false, as the concatenation sometimes winds up renaming the methods.
  • Instead of relying on Webpack’s default configuration for Terser, supply (via .optimization.minimizer) an instance of terser-webpack-plugin configured to preserve the calls and comments.
    • .terserOptions.mangle.reserved set to reserve the four methods.
    • .terserOptions.format.comments set to a callback that identifies translator comments.
    • .extractComments set to a callback that identifies the license comments Terser preserves by default, which will now be extracted to a separate file instead to reduce the size of the bundle.
  • We also included Calypso’s @automattic/babel-plugin-preserve-i18n plugin to further help preserve the i18n method names.

Solution, part 2: To address the “even if it’s working now, it might break later” problem, and to help identify coding patterns that can break the i18n method calls even with the above configuration, we created @automattic/i18n-check-webpack-plugin. This plugin extracts the strings from the original sources and the output bundle to compare them and see if anything seems to have gone missing, so if something breaks it’ll make the build fail instead of having to wait for someone to notice the broken i18n and report it.

The documentation for the check plugin includes some known problematic code patterns and fixes for them.

Problem 3: Lazy-loaded Webpack bundles

For “entry” bundles, Webpack expects your HTML to include any additional files (e.g. CSS extracted by mini-css-extract-plugin) yourself. In a WordPress plugin this is fairly straightforward to do from PHP (and we made it even easier for ourselves using automattic/jetpack-assets as described above), and that includes loading of the appropriate translation data into @wordpress/i18n.

But if you use code like import( /* webpackChunkName: "async" */ './something' ), Webpack will create a “lazy-loaded” bundle that isn’t loaded until that import() call is executed. In Jetpack we have one of these in the Instant Search module. For such lazy-loaded bundles the Webpack runtime knows how to load the extracted CSS, but it knows nothing about WordPress translation data. 😬

When we looked around we saw that Calypso had a fairly complicated solution in their code, a generic hook added in the Webpack runtime and specific code to load data when that hook fired. Woo had tried to adapt that but gave up in favor of tricking WordPress into loading the lazy bundle’s translations non-lazily. Neither solution appealed.

Solution: We created @automattic/i18n-loader-webpack-plugin to teach Webpack how to load the WordPress translation data. It’s designed to work in concert with @wordpress/dependency-extraction-webpack-plugin and automattic/jetpack-assets: when i18n-loader encounters a bundle that can lazy-load other bundles that use @wordpress/i18n, it will register a dependency on a “@wordpress/jp-i18n-state” module via the former that’s provided by the latter. The state data lets the Webpack runtime inside the bundle know how to locate the translation data, which it will then download and register with @wordpress/i18n during the lazy-loading process.

Problem 4: Text domains in shared React components

As part of the “Jetpack RNA” project, we’ve begun creating React components that can be shared by multiple plugins, like our own Jetpack Backup and Jetpack Search plugins.

But remember how the __() call needs to specify a domain, which is supposed to be a constant string (not a variable) and must match the plugin’s slug? 😬

Solution: We created @automattic/babel-plugin-replace-textdomain, a simple Babel plugin to rewrite the domains as the components are being bundled.

Problem 5: Bundled Composer packages being skipped

WordPress core doesn’t really use Composer; they have a composer.json, but just to pull in PHPUnit and a few other development tools. Where WordPress core needs libraries at runtime, they copy them in statically. Plugins either do the same or include Composer’s vendor/ directory in the code checked into WordPress.org SVN.

The WordPress.org translation infrastructure assumes that anything in vendor/ either has no translations or has its own translation mechanism entirely, instead of intending to use WordPress’s. (Although since __() and such would be defined by WordPress rather than the plugin, I’m not sure how that’s intended to work.) In our case, we do really want these packages’ strings included in the plugin’s language pack. 😬

Solution: Composer allows for custom installer plugins, which can install packages into different locations based on the “type” field in the package’s composer.json. We created automattic/jetpack-composer-plugin that installs “jetpack-library” packages into jetpack_vendor/, and set the types of the relevant packages to “jetpack-library”.

Problem 6: Text domains in Composer packages

As with the shared React components, the bundled Composer packages need to be using the plugin’s text domain because that’s where the translations are going to be. And this time we’re not compiling them into a bundle, so a compile-time replacer wouldn’t work. 😬

Solution: The solution here comes in several parts.

  1. We have automattic/jetpack-composer-plugin write an “i18n-map.php” file into jetpack_vendor/, collecting the WordPress plugin’s slug (set in its composer.json) and each package’s textdomain and version (from their composer.jsons).
  2. The WordPress plugin passes that file to automattic/jetpack-assets, which determines the mapping from each package’s domain to an appropriate plugin’s.
  3. Assets hooks into __() and such to try the plugin’s domain if no translation was found for the package’s. It also hooks into the script translation file loader to point to the script translation files included with the plugin’s language pack instead of the nonexistent packs for the packages’ text domains. And finally it includes the mapping in the state data for @automattic/i18n-loader-webpack-plugin so that can load the correct file for any lazy-loaded bundles.

We also made sure that our monorepo’s CI checks would catch common cases where developers might wind up with wrong text domains, using existing linting rules from @wordpress/eslint-plugin and wp-coding-standards/wpcs and custom checks to verify those rules’ configurations are in sync with each other and with composer.json.

If WordPress Core were to take on this problem, I think they could do it a bit better:

  1. Switch the translation infrastructure from being based on plugins and themes (which all use the plugin or theme slug as the text domain) to being based on the text domains directly. For example, instead of https://api.wordpress.org/translations/plugins/1.0/ and https://api.wordpress.org/translations/themes/1.0/ just have one endpoint that takes the domain.
  2. Let code declare to WordPress which domains it needs beyond the defaults of “plugin slug” and “theme slug”. This is so WordPress can download those extra domains, I don’t think WordPress cares beyond that.
  3. Let us register projects that aren’t plugins or themes (e.g. our packages) on translate.wordpress.org, so the packages can be translated and WordPress can fetch those translations like it does plugins and themes.

That way the plugin only needs to declare the packages’ text domains, and the translators would only have to translate each package’s strings once instead of doing so for every plugin using the package.

Summary and conclusion

We created several pieces to make everything work:

We also took advantage of @wordpress/dependency-extraction-webpack-plugin and Calypso’s @automattic/babel-plugin-preserve-i18n, as well as linter rules from @wordpress/eslint-plugin and wp-coding-standards/wpcs (and a fork of phpcs that we’ve been trying to upstream to let us have per-directory configs) to help developers in our monorepo keep text domains straight.

Overall this was quite a bit of work, but Jetpack’s i18n is now better than ever before. And we hope that this post describing the problems we found and our solutions might help other plugin developers improve their i18n as well.

Automatically Deploy to WordPress.com Using GitHub Actions

WordPress.com provides you with a great experience out-of-the-box, from superb SEO to social media sharing, custom web fonts, and so much more.

At WordPress.com, we also cover very advanced needs, such as connecting GitHub to sites on a Business or eCommerce plan. This allows you to automate deployment using GitHub Actions, which can become a huge productivity boost for developers. You’ll be happy to know that GitHub and WordPress.com work beautifully together.

Prerequisite: Start your WordPress.com site now

Tutorial Overview, GitHub Deployment Automation

This is a tutorial that explains how to achieve deep customization of your WordPress.com site using SFTP access, including how to deploy custom themes and plugins. Moreover, we’ll explain how you can automate the deployment of those changes by simply connecting your GitHub repository to WordPress.com. Whenever you push changes to a specific branch of your GitHub repository, those changes will automatically be deployed to your site at WordPress.com as well.

Create SFTP Credentials at WordPress.com

At WordPress.com, create SFTP Credentials for your site. SFTP access is available to administrators on the WordPress.com Pro plan.

WordPress.com SFTP creds settings page

Log In Over SFTP, Manually Review Directory Structure

In the screenshot below, which was taken from Filezilla (a popular SFTP client), you can see the primary extensibility features in WordPress, i.e., themes and plugins. Themes are in /wp-content/themes; plugins are in /wp-content/plugins.

If you recreate this structure in a Git repository starting from the /wp-content directory, then whenever you push changes to the GitHub repository, they will be deployed automatically to your site at WordPress.com.

Some core directories and files have a question mark (?) icon next to them. This denotes they are part of your site’s core installation. We do not allow modifying core files as they are required for your site to function properly.

Filezilla SFTP client example screenshot

Create GitHub Repository

I’ve created an example GitHub repository named: wp-github-actions-test-site. Inside this repository, I’ve added a custom theme and a custom plugin that I’ll deploy automatically as part of this tutorial. I have already pushed these changes to the main branch of my example repository. Using my example (screenshot below), please create a GitHub repository for your site at WordPress.com.

Creating GitHub repo example screenshot

Create New Branch: wordpress.com

Create a new branch with the name wordpress.com, such that you have two branches: main and wordpress.com. Whenever you push changes to the wordpress.com branch, you want those changes to be deployed automatically to WordPress.com. You can create the new branch using GitHub’s interface, as seen below, or from the command line. The choice is yours.

GitHub repo branch structure example screenshot

Create GitHub Action

On the main branch, create a workflow by copying and pasting the code snippet (provided below). Please copy & paste the snippet with no modifications into .github/workflows/wordpress.com.yml

Note: It is easiest to use GitHub’s interface for this, as shown here.

Creating GitHub Action workflow example screenshot
GitHub Actions WordPress automation example workflow script screenshot

WordPress.com Workflow Code Snippet

Copy & paste into .github/workflows/wordpress.com.yml

name: Deploy to WordPress.com
    branches: [ wordpress.com ]
    name: FTP-Deploy-Action
    runs-on: ubuntu-latest
      - uses: actions/checkout@master
          fetch-depth: 2
      - name: FTP-Deploy-Action
        uses: Automattic/FTP-Deploy-Action@3.0.1
          ftp-server: sftp://sftp.wp.com/htdocs/
          ftp-username: ${{ secrets.FTP_USER }}
          ftp-password: ${{ secrets.FTP_PASSWORD }}
          known-hosts: "\
            sftp.wp.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB\

Create GitHub Secrets

The workflow code snippet references two secrets:

- ${{ secrets.FTP_USER }}
- ${{ secrets.FTP_PASSWORD }}

Create these two secrets in your GitHub repository using the SFTP credentials that you obtained at WordPress.com in an earlier part of this tutorial. Please define both the FTP_USER and FTP_PASSWORD secrets.

GitHub Actions workflow secret example screenshot
GitHub Actions secrets settings in workflow

Push Changes, Test Deployment Automation

Now that you have everything configured properly, update your local copy of the GitHub repository using Git from the command line. Start by fetching all updates, then pull changes into the main branch, merge them into the wordpress.com branch, and finally push those changes to the wordpress.com branch. In doing so, you will test the GitHub Action workflow to make sure things are working properly.

$ git fetch --all
$ git checkout main && git pull
$ git checkout wordpress.com && git merge main
$ git push --set-upstream origin wordpress.com

At this point, you can visit the Actions tab at GitHub to review and observe status. Since you pushed to the wordpress.com branch, the workflow you created should already be running. When it’s finished, the status icon will turn green with a checkmark indicating your changes were deployed successfully to WordPress.com.

GitHub Actions workflow ran success screen within repository

In this case, the workflow is deploying your last change, which was to “Create the wordpress.com.yml” file itself — from a previous step in this tutorial.

In the future, when you merge and then push any changes, such as theme or plugin updates, from the main branch into the wordpress.com branch, they’ll be deployed to WordPress.com automatically. Yay! 🎉

The Pro Plan at WordPress.com

At WordPress.com, you’ll be well-equipped without installing custom themes or plugins. However, if you want to take your site to the next level, our Pro plan supports several additional features. This plan includes increased storage, custom themes, custom plugins, and SFTP access, making the GitHub integration described in this tutorial possible.

Start your WordPress.com site now

The Rise of WordPress Frontend Options, Themes vs. REST API + JS/React: Decoupling WordPress

The opportunity to move to a decoupled or headless WordPress system is an emerging discussion because of the broad range of possibilities it opens for developers with complex architecture needs. Decoupling, or separating the backend from the frontend with a framework like Strattic, also delivers a level of complexity that should be factored in when planning a project’s development. Certain projects can benefit from decoupling, while the fully integrated WordPress platform better serves most use cases.

When the words “Learn JavaScript Deeply” were spoken by Matt Mullenweg in 2015, the world hadn’t yet fully realized the scope of what would soon be possible. Then, when the WordPress REST API was released as part of core in 2016, the platform made a giant leap into the future. Of course, Gutenberg, the WordPress editor introduced in late 2018, highlighted what could be accomplished with this new feature as it uses the REST API to create and update posts by communicating with the WordPress core backend. Both the REST API and a modern JS framework built on React are shipped with every copy of WordPress and are now integral components.

This evolution of WordPress means that today, developers are no longer limited to PHP. Instead, developers can create apps built on React and other JavaScript frameworks, served by Node, and powered by the WordPress REST API. This enables WordPress development by frontend developers who don’t write PHP, as well as integration between WordPress and other frontend technologies and systems. 

Key Issues To Consider 

First, a reminder of some key issues worth seriously considering as you weigh decoupling, versus using WordPress as a complete solution. 

  • Performance – Many developers want to decouple in the belief that this increases performance or scale. This, however, rarely proves itself in reality.  Decoupled front-ends must be scaled separately from the WordPress backend. A good host largely solves scaling in traditional architectures. A top-tier enterprise WordPress platform, such as WordPress VIP, will also scale endpoints for decoupled architectures and can serve Node apps at the same scale as anything else being served.  However, decoupled front-ends present their own points of failure, their own caching mechanisms, and generally increase performance risk.  For example, if the site is down, is the front-end causing the problem?  If so, which part of it?  Or is it the API?  
  • Total Cost of Ownership (TCO) – Even teams with mature JS capacity typically find that decoupled experiences have higher TCO than traditional ones. This is primarily due to duplicate hosting, multiple technologies, split teams, multiple codebases, multiple caching strategies, and more. The overall cost of support and build of a decoupled project is almost always higher than a traditional integrated WordPress build. It is essential to consider cost in terms of the whole lifecycle of the project.
  • Solved problems / ecosystem – Out-of-the-box WordPress architectures have more access to integrations and community solutions than custom front-ends.  Sure, React supports many community extensions and integrations, but when we limit the scope to the kinds of extensions required for content apps, WordPress’ extensions and integrations are almost always easier and cheaper to figure out. Examples include AMP and other delivery channels, authentication/SSO, many platform integrations, and even features such as “preview” become an issue that needs to be solved anew for decoupled sites.

Available Toolkit

Today’s decoupled WordPress toolkit is quickly growing, and it includes the following popular solutions.

  • GraphQL / WPGraphQLGraphQL is a Query Language built to craft more precise and lightweight queries to bring only the data you need from the REST API into your front-end application. WPGraphQL extends this framework to be specific to WordPress.
  • GatsbyJS – GatsbyJS is a React and GraphQL-powered static site generator built for speed.
  • FrontityFrontity is a React framework for WordPress. Out of the box, it is set up to consume content from the WordPress REST API, giving developers a big head start by providing many of the most common queries already built-in.
  • StratticStrattic is a static site generator and hosting platform tuned for WordPress.

When Decoupling is an Option

Some of the most relevant reasons developers choose a decoupled approach for complex WordPress projects include:

  • Portability – The app or digital experience needs to be extremely portable or agnostic of the back-end technology.
  • Diverse data sources – The app or digital experience pulls data from more than a small handful of sources, with WordPress being only one of those, and where the WordPress source is probably not the most central of those sources.
  • Leveraging an existing JS-oriented dev team – Frequently, teams of JS developers, especially newly minted ones, want to build Node apps and leverage modern front-ends like React. While the current team’s capabilities are worth considering, this reason may or may not be outweighed by other factors. In addition, there is plenty of exciting work for modern JS developers to do within WordPress itself.
  • Moving into the future of modern JS frameworks – Sometimes, teams want to leverage modern frameworks because the technology is newer, supports app-like behaviors natively in the browser, and it is relatively easy to hire qualified modern JS devs.  These are valid considerations, especially if there are other business reasons to invest in this capacity. However, it is crucial to avoid automatically assuming that modern JS does everything better, cheaper, or faster than WordPress’s mix of PHP and modern JS.

AccuWeather: Making Brilliant Use of Decoupling

This marriage of technologies is the result of leveraging the long-term stability of WordPress with its vast ecosystem with modern frameworks such as React. When used appropriately, this union enables the creation of brilliantly complex applications. One example of this in use is the extremely popular AccuWeather. 

AccuWeather is recognized as the most accurate source of weather forecasts and warnings in the world, serving roughly 1.5 billion people daily. The system comprises weather forecasts, local media partnerships, enterprise solutions, APIs, news, videos, podcasts, and weather data. 

AccuWeather uses a decoupled WordPress stack that includes over 750+ million WordPress requests per day and 50+ billion data calls per day. 

So what advantages were gained by AccuWeather moving to this architecture? 

  1. By retaining the use of WordPress, they can utilize the backend UI/UX experience for the content team. While content is only one part of AccuWeather, it is an ever-growing piece of the brand and company. Using the WordPress backend, the editorial team can work with the most widely adopted and accessible CMS platform, which is used by 40% of the web.
  2. By decoupling the front-end from the back-end, rapid site redesign was made possible without worrying about making massive changes to the platform’s CMS side. 
  3. Finally, the decoupling makes the content easily accessible by other applications. Various development teams, such as app or enterprise teams, can present the content anywhere. For example, they can include a newsfeed with localized content in AccuWeather apps or present targeted video in a widget on a partner site. 

While AccuWeather makes extensive use of the REST API, many other companies and projects can benefit from utilizing and extending the REST API. For example:

  • A vendor may utilize the API for large-scale content migration.
  • A CMS dev team may extend the API to to create mobile apps, or facilitate communication that helps complete 3rd-party integrations.
  • Much like Laravel, WordPress offers developers an extensible REST API framework. A company can easily add new routes and endpoints, giving them a more predictable and structured way to interact with site content or other backend data. They’ll spend less time building tools to access data, and more time creating better user experiences.

One Final Consideration

If all of this seems like the perfect solution for every future project, there’s one final thing to consider before making that jump. 

Decoupling requires a developer or team of developers throughout the entire lifecycle – Developers need to not only be involved in the creation of the project, but will also be needed to manage the site throughout its lifecycle. The client will rarely be able to manage this on their own. While many fully integrated WordPress websites can be left in the care of the site owner, a decoupled WordPress site is usually far too complex for the average site owner to maintain, keep updated, and ensure security and forward compatibility.

Decision Worksheet

Use an internal decision worksheet, such as the one we’ve provided here, to help determine if your project should move forward with a decoupled architecture or not. The more times you answer “No” to the questions, the more caution you should exercise before moving forward. You may still decide that decoupled is the best option, even if you’ve answered ‘No’ to many of the questions, but more deliberation might be needed in that instance.

The Best Choice For You

WordPress leverages the best of a stable, mature product with the promise of evolving with ever-changing modern technological future platforms. Some projects may benefit from the WordPress platform’s integrated approach, while others might require a decoupled architecture. When making decisions, consider the following:

  1. Ask why – Identify the key drivers of your decision, and be honest about this. Are you choosing the more complex decoupling method because it’s the latest cool technology, or is it because it best serves the project?
  2. Consider the total cost of ownership – The TCO over the whole project lifecycle, including multiple hosting, increased complexity, more technical debt, etc., may be significantly higher than you expect.
  3. Look ahead – The landscape around JavaScript and WordPress is changing and expanding rapidly. Can you position your team and your application to thrive in the years and months to come?
  4. Embrace JavaScript – Regardless of your decision to decouple or not, embracing JavaScript will enable you to do exciting things, even within WordPress internally. Learn JavaScript Deeply…still as true today as the day Matt first spoke those words.

Many factors will influence your decision, but WordPress will always be there, either integrated or decoupled, to move your project forward in the best possible way.

Written in collaboration with Donna Cavalier & Jason Caldwell from WordPress.com, and Ryan Sholin & Matt Perry and WordPress VIP.