Automatically Deploy to WordPress.com Using GitHub Actions

WordPress.com provides you with a great experience out-of-the-box, from superb SEO to social media sharing, custom web fonts, and so much more.

At WordPress.com, we also cover very advanced needs, such as connecting GitHub to sites on a Business or eCommerce plan. This allows you to automate deployment using GitHub Actions, which can become a huge productivity boost for developers. You’ll be happy to know that GitHub and WordPress.com work beautifully together.

Prerequisite: Start your WordPress.com site now

Tutorial Overview, GitHub Deployment Automation

This is a tutorial that explains how to achieve deep customization of your WordPress.com site using SFTP access, including how to deploy custom themes and plugins. Moreover, we’ll explain how you can automate the deployment of those changes by simply connecting your GitHub repository to WordPress.com. Whenever you push changes to a specific branch of your GitHub repository, those changes will automatically be deployed to your site at WordPress.com as well.

Create SFTP Credentials at WordPress.com

At WordPress.com, create SFTP Credentials for your site. SFTP access is available to administrators on the WordPress.com Business and eCommerce plans.

WordPress.com SFTP creds settings page

Log In Over SFTP, Manually Review Directory Structure

In the screenshot below, which was taken from Filezilla (a popular SFTP client), you can see the primary extensibility features in WordPress, i.e., themes and plugins. Themes are in /wp-content/themes; plugins are in /wp-content/plugins.

If you recreate this structure in a Git repository starting from the /wp-content directory, then whenever you push changes to the GitHub repository, they will be deployed automatically to your site at WordPress.com.

Some core directories and files have a question mark (?) icon next to them. This denotes they are part of your site’s core installation. We do not allow modifying core files as they are required for your site to function properly.

Filezilla SFTP client example screenshot

Create GitHub Repository

I’ve created an example GitHub repository named: wp-github-actions-test-site. Inside this repository, I’ve added a custom theme and a custom plugin that I’ll deploy automatically as part of this tutorial. I have already pushed these changes to the main branch of my example repository. Using my example (screenshot below), please create a GitHub repository for your site at WordPress.com.

Creating GitHub repo example screenshot

Create New Branch: wordpress.com

Create a new branch with the name wordpress.com, such that you have two branches: main and wordpress.com. Whenever you push changes to the wordpress.com branch, you want those changes to be deployed automatically to WordPress.com. You can create the new branch using GitHub’s interface, as seen below, or from the command line. The choice is yours.

GitHub repo branch structure example screenshot

Create GitHub Action

On the main branch, create a workflow by copying and pasting the code snippet (provided below). Please copy & paste the snippet with no modifications into .github/workflows/wordpress.com.yml

Note: It is easiest to use GitHub’s interface for this, as shown here.

Creating GitHub Action workflow example screenshot
GitHub Actions WordPress automation example workflow script screenshot

WordPress.com Workflow Code Snippet

Copy & paste into .github/workflows/wordpress.com.yml

name: Deploy to WordPress.com
on:
  push:
    branches: [ wordpress.com ]
jobs:
  FTP-Deploy-Action:
    name: FTP-Deploy-Action
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 2
      - name: FTP-Deploy-Action
        uses: Automattic/FTP-Deploy-Action@3.0.1
        with:
          ftp-server: sftp://sftp.wp.com/htdocs/
          ftp-username: ${{ secrets.FTP_USER }}
          ftp-password: ${{ secrets.FTP_PASSWORD }}
          known-hosts: "\
            sftp.wp.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB\
            AAABAQDwfT/YEhOKO2Z0+XrjRqUS5Q2Ali6AlhOhZtzlIfMOvm03Sype\
            DJH70tlUHasS+nm0SnZ01fOiEeAXa91ZhMihIYUT3nTGuiA2J3uVYsyS\
            CJefvhWc0kg1FbEus3V3cVmx4e3XctdkzLbOgPNngypZocbP+8yCpbx6\
            Kb9lihmgTjgGn2QzbK1enRSzsN/CbjVhej9jwukbrWqdCrQsKAsoZ2p6\
            YCtcKbHS+Yy4RwcO9PxZUBkeMXUrejms027bRcdVfwf55hWSD9xYEHpE\
            HupkSL4ofWs3UKeRGz+jCCzl7Nu0S6VSwK4Zzll0auHI0Hwh8WKTJbSn\
            1gxCdF93/rmP\
          "

Create GitHub Secrets

The workflow code snippet references two secrets:

- ${{ secrets.FTP_USER }}
- ${{ secrets.FTP_PASSWORD }}

Create these two secrets in your GitHub repository using the SFTP credentials that you obtained at WordPress.com in an earlier part of this tutorial. Please define both the FTP_USER and FTP_PASSWORD secrets.

GitHub Actions workflow secret example screenshot
GitHub Actions secrets settings in workflow

Push Changes, Test Deployment Automation

Now that you have everything configured properly, update your local copy of the GitHub repository using Git from the command line. Start by fetching all updates, then pull changes into the main branch, merge them into the wordpress.com branch, and finally push those changes to the wordpress.com branch. In doing so, you will test the GitHub Action workflow to make sure things are working properly.

$ git fetch --all
$ git checkout main && git pull
$ git checkout wordpress.com && git merge main
$ git push --set-upstream origin wordpress.com

At this point, you can visit the Actions tab at GitHub to review and observe status. Since you pushed to the wordpress.com branch, the workflow you created should already be running. When it’s finished, the status icon will turn green with a checkmark indicating your changes were deployed successfully to WordPress.com.

GitHub Actions workflow ran success screen within repository

In this case, the workflow is deploying your last change, which was to “Create the wordpress.com.yml” file itself — from a previous step in this tutorial.

In the future, when you merge and then push any changes, such as theme or plugin updates, from the main branch into the wordpress.com branch, they’ll be deployed to WordPress.com automatically. Yay! 🎉

Business and eCommerce Plans at WordPress.com

At WordPress.com, you’ll be well-equipped without installing custom themes or plugins. However, if you want to take your site to the next level, both our Business and eCommerce plans support several additional features. These plans include increased storage, custom themes, custom plugins, and SFTP access, making the GitHub integration described in this tutorial possible.

Start your WordPress.com site now

State of WordPress.com Elasticsearch Systems 2016

We get asked periodically about how extensively we are using Elasticsearch. And it has come up twice in the past week, so time to write a blog post.

We are constantly expanding what we are using Elasticsearch for and so although some previous posts have broadly define what we are doing, they don’t really capture the continually expanding scale.

So here are some quick bullet points about what we currently have deployed:

  • Five clusters with a mix of versions:
    • 42 data nodes spread across 3 US data centers running ES 1.3.4. This cluster mostly runs related posts queries. 1925 shards. 11B docs. 43TB of data. 60m queries/day. 12m index ops/day (has been as high as 940m in a day though). Each index is 175 shards and has 10m blogs in it. Each blog is routed to a single shard so almost all queries only hit one shard, but we can (and do) search across multiple shards for some use cases.
    • 6 data nodes across 3 DCs running ES 1.3.9. Hosts our WordPress.com VIP indices and lots of other use cases. 321 indices (mostly VIPs). ~8m queries/day. ~1.5m index ops/day. Typical VIP index config is a single shard that is replicated across the three data centers. Most of these indices are small enough that sharding would reduce performance and reduce query relevancy.
    • 12 data nodes across 3 DCs running ES 1.7.5. Primarily powers search.wordpress.com. Indexes the past 6 quarters of all posts. One index per quarter with 30 shards per quarter. Queries typically hit all 180 shards.
    • 3 data nodes across 3 DCs running ES 2.3.1. Currently an experimental cluster as we work to migrate to 2.x. Only production index right now is for en.support.wordpress.com.
    • 15 (and possibly expanding to 100) data nodes for a Logstash cluster running ES 2.3. A lot of logging use cases for many different services. Growing rapidly.
  • All of our clusters use three dedicated master nodes with one master in each data center. The first cluster has its own master nodes. The next three share master servers with multiple instances of ES running on each server.
  • Typical data server config:
    • 96GB RAM with 31GB for ES heap. Remaining gets used for file system caching
    • 1-3 TB of SSD per server. In our testing SSDs are very worthwhile.
  • Query speed:
    • Related Posts: median 44ms; 95th percentile: 190ms; 99th percentile: 650ms. This is way lower than when we launched in 2013 and 99th percentile was 1.7 seconds.
    • VIP Queries: median: 25ms; 95th percentile: 109ms; 99th percentile: 311ms
    • search.wordpress.com queries: median: 130ms; 95th percentile: 250ms; 99th percentile: 260ms
  • Client-side Optimizations:
    • We cache all queries results in memcache which cuts our ES query rate in half
    • memcache timeouts vary from 30 seconds to 36 hours depending on use case
    • We analyze all queries on the client side and optimize the ES filters:
      • have a blacklist of fields that we never cache (blog_id, post_id, author_id) because they have such high cardinality (100m+ unique ids)
      • we rewrite and/or/not filters into bool queries and try to flatten them into a single filter
      • We don’t allow some types of queries (we have a whitelist)
      • We don’t allow facets/aggregations on certain fields (content, title, excerpt)
    • We generally don’t allow paging too deep or returning thousands of results at once
    • A general pattern we use is to use ES to get IDs for content, and then we get the real content from MySQL for displaying to users. This reduces what data ES needs (we strip out HTML), and we can be certain the data is not out of date since ES can be up to 60 seconds out of date in some cases (though typically is less than 5 seconds).
  • Query Use Cases (in order of query frequency):
    • Related Posts
    • Replacing WP_Query calls by converting slow SQL calls to an ES query (WordPress tag/category pages, home pages, etc)
    • search.wordpress.com
    • Language Detection using ES langdetect plugin (used for every post we index)
    • Analyze API (used to perform reliable word counting regardless of language – in conjunction with the langdetect call)
    • Blog Search (replacing the built in WordPress site search)
    • Theme Search
    • Search Queries that are used when reindexing content (eg when a blog’s tag is renamed we need to search for all posts with that tag and reindex them)
    • Various support searches
    • A number of custom VIP use cases
    • A number of custom internal use cases (searching our p2s, suggesting posts that may be relevant to read, searching our internal docs, etc)
    • Calypso /posts and /pages for getting/searching all posts a user has authored across all their blogs (potentially hundreds)
  • ES Plugins Deployed:
    • Whatson for looking at shard distribution, disk usage, index size, etc
    • StatsD for performance monitoring (we also send StatsD data from the client about query speed) See the screenshots of dashboards below.
    • ICU Analysis
    • Langdetect
    • SmartCN and Kuromoji Analyzers
    • Head

 

Since images are always fun, here are our Graphana dashboards for our largest cluster over the past 6 hours. The first is our client-side tracking of query/indexing/etc speed

Screen Shot

Second is our aggregated stats (from the StatsD plugin) about the cluster’s performance:

Screen Shot

This cluster/index has been really solid for us over the past two years since it was last built. We have some known issues that have us stuck on 1.3.4, but we’ve also had times where the cluster went many months without any incidents. In general the incidents we have seen have been caused by external factors (usually over indexing or some other growth in the data).

 

 

 

WordPress.com for Windows Metro and the REST API

WordPress.com for Windows Metro

When we were starting work on the WordPress.com for Windows Metro app, we decided to build it entirely with the brand new WordPress.com REST API. The API gave us everything we needed to create a full-featured application that includes reading blogs, social features (post likes, reblogs), and creating new posts. Let’s take a look at a few code samples to see how we leveraged the API:

*note* All sample code is a condensed version of code from the Metro App. You can view the full source of the app at our trac site.

Creating a New Post

Once a user has authorized the app using OAuth (more info here), they can create new posts in the app. The /posts/new endpoint makes it very easy to create a new post via an XMLHttpRequest (xhr):


function publishPost() {
var title = "I Need a Good REST";
var content = "That's probably the corniest blog title ever created...";

var data = new FormData();

data.append('title', title);
data.append('content', content);
data.append('format', 'standard');

//note: blogID and users_access_token are received from the OAuth API
var url = 'https://public-api.wordpress.com/rest/v1/sites/' + blogID + '/posts/new';
 WinJS.xhr({
     type: 'POST',
     url: url,
     headers: { 'Authorization': 'Bearer ' + users_access_token },
     data: data
   }).then(function (result) {
     var resultData = JSON.parse(result.responseText);
  if (resultData.ID) {
    //post published!
  }
  else {
    //post publish failed
  }
}

Liking a Post

The Windows Metro app takes full advantage of the social features on WordPress.com, including likes, blog follows, and reblogs. Let’s take a look at using the /likes endpoint so that we can like a post through the API:


var curText = m.target.innerText,
likeButton = document.getElementById('like');
//just an example post_id
var post_id = 123;
//note: blogID and users_access_token are received from the OAuth API
if (curText == 'Like') {
   label.innerText = 'Liking...';
   url = 'https://public-api.wordpress.com/rest/v1/sites/' + blogID + '/posts/' + post_id + '/likes/new';
 } else {
   label.innerText = 'Unliking...';
   url = 'https://public-api.wordpress.com/rest/v1/sites/' + blogID + '/posts/' + post_id + '/likes/mine/delete';
 }

WinJS.xhr({
   type: 'POST',
   url: url,
   headers: { 'Authorization': 'Bearer ' + users_access_token }
 }).then(function (result) {
   //like operation successful!
 }, function (result) {
   //error
 });

You can see how we can also remove the like on the post by simply changing the endpoint on the post to /likes/mine/delete.

Conclusion

These examples are a small drop in the bucket of what you can do with the WordPress.com REST API. For a full list of the endpoints that are available, check out the documentation.

We’d love to hear how you are using the API in your applications! Drop us a note at our contact form to get in touch with the developer.wordpress.com team.