WordPress REST API on WordPress.com

As you might know if you follow WordPress developer news, the WordPress REST API content endpoints are slated to be merged into WordPress core in the upcoming 4.7 release. These endpoints cover basic WordPress features (posts, pages, custom post types, revisions, media, comments, taxonomies, users, and settings). While they have also been available via the WP REST API plugin for some time, this is the latest step in a multi-year project to bring a modern, RESTful API to all WordPress installations.

We are pleased to announce that WordPress.com is now running the WordPress REST infrastructure alongside our pre-existing v1 API. Additionally, we have made the new content endpoints available now so that developers can try them out, provide feedback and drive wider adoption of the new API.

The success metrics for this feature, detailed here, focus on increasing utilization of the new API in plugins, themes, third-party clients and within WordPress core itself. This new API represents the future of WordPress and WordPress.com, and we want to make it as solid as we can. We encourage all WordPress developers to take it for a spin and let us know what you think!

Using the New API

The WordPress REST API is available on WordPress.com with the following base URL: https://public-api.wordpress.com/wp/v2/. This new API is backwards-incompatible with the v1 API, which we have no plans to deprecate at this time.

Going forward, we encourage developers to write new code against the v2 API whenever possible. This has the added benefit of making your applications able to run against both WordPress.com and self-hosted WordPress sites.

Documentation for the WordPress REST API and its endpoints can be found at http://v2.wp-api.org/.  Our installation of the API uses the same OAuth2 authentication tokens as our version 1 API.

To make testing and exploration easier, we’ve also updated our API console to support both versions of our API. You can switch API versions using the dropdown toggle in the upper left:

wpcom-wp-rest-api-1

 

Once you’ve selected the WP REST API, you can view and select endpoints and make authenticated requests just as in the previous version:

wpcom-wp-rest-api-2-cropped

Caveats

For self-hosted sites running the Jetpack plugin, we expect read requests to work (but you must be running at least Jetpack version 4.3.2).  Write requests to Jetpack sites do not work yet, since this will require the site to be running the content endpoints and a future version of the Jetpack plugin that supports them.

In addition, a few endpoints are restricted or disabled on WordPress.com (generally due to security concerns and/or parity with our existing API). A few examples:

  • Listing and retrieving media items requires authentication
  • Listing and retrieving user info requires authentication
  • Endpoints to create/edit/delete users are disabled
  • Updating the email and URL settings via the settings endpoint is disabled

Examples

Here are some examples that could serve as a starting point. If you build your own, please share it in the comments!

How to Provide Feedback

Improved REST API Support for Custom Taxonomies

The WordPress.com REST API has always enabled developers to manage categories and tags for the posts on a site. Until now, however, if a site had enabled any custom post types with custom taxonomies, developers were out of luck if they had hoped to manage custom terms using the REST API.

We’re happy to announce that this is no longer the case, with improved support for custom taxonomies. This includes a handful of new endpoints and enhancements to existing endpoints.

Specifically, you can now…

Note that taxonomies and terms are only included in a response if the taxonomy is registered as public or if the authorization token sent with the request has the proper capabilities for viewing or managing that taxonomy. Refer to the register_taxonomy function documentation for more information.

As an example, if a Jetpack site supported a custom book post type with a genre taxonomy, I could create a new book by issuing the following cURL command:

curl \
 -H 'authorization: Bearer YOUR_API_TOKEN' \
 --data-urlencode 'title=Sample Book' \
 --data-urlencode 'type=book' \
 --data-urlencode 'terms[genre][]=Fiction' \
 'https://public-api.wordpress.com/rest/v1.2/sites/example.com/posts/new/'

If you’re new to working with the WordPress.com REST API, we recommend you head on over to our Getting Started with the API guide to learn more.

Introducing our WordPress.com Automated e2e Tests

At WordPress.com we strive to continuously deliver new functionality whilst ensuring that we provide a consistent and reliable user experience.

One key element to all of this is developing self-testing code; that is, creating and maintaining a comprehensive suite of automated tests that are developed in conjunction with our functional code.

The WordPress Calypso User Interface supports React unit and component testing. Because the WordPress.com user experience depends on many other factors beyond just the precision of our Calypso components, we have also been building a suite of automated end-to-end (e2e) tests that verify our critical user flows are working as intended.

Now, the WordPress.com team is proud to announce that we have released our automated e2e test suite as open-source. You can find it here on GitHub.

Why open source is so important

You could say our automated testing pyramid looks like this:

WordPress.com Automated Testing Pyramid

The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.

~ Matt Mullenweg, Automattic CEO

The technical details

Our tests use the ES2015 syntax on Node, which should make it it easy for anyone familiar with Calypso to contribute.

We are utilizing the open-source selenium-webdriver project to drive real user browsers. We write our e2e tests in the same test specifications library as Calypso (Mocha) and we run these on the same CI platform that being on headless CircleCI Linux containers.

Since our platform is fully responsive, we run the same tests (with slightly different logic) against three different screen resolutions to ensure our customer experience functions across different device types.

We’ve also done some neat things like developing real-time Slack notifications (with screenshots) when tests are failing so we can investigate quickly.

It’s still early days, and we’re still striving to make these tests as deterministic and reliable as possible.

You can run these tests against a local WordPress Calypso development environment. We automatically run these tests on every WordPress.com deployment, as well as every six hours, to ensure end-to-end flows are working as intended for our users. We are planning to run these, or a subset of these, against pull requests as they are developed to prevent bugs being introduced into our platform.

Automated tests have already been successful in making us aware of issues sooner so we can fix them faster.

Get involved!

Feel free to check out our repository, make a fork, and provide us with feedback or suggestions. Pull requests are always welcomed.

***

Alister Scott is an Excellence Wrangler for Automattic and blogs regularly about software testing at WatirMelon.

State of WordPress.com Elasticsearch Systems 2016

We get asked periodically about how extensively we are using Elasticsearch. And it has come up twice in the past week, so time to write a blog post.

We are constantly expanding what we are using Elasticsearch for and so although some previous posts have broadly define what we are doing, they don’t really capture the continually expanding scale.

So here are some quick bullet points about what we currently have deployed:

  • Five clusters with a mix of versions:
    • 42 data nodes spread across 3 US data centers running ES 1.3.4. This cluster mostly runs related posts queries. 1925 shards. 11B docs. 43TB of data. 60m queries/day. 12m index ops/day (has been as high as 940m in a day though). Each index is 175 shards and has 10m blogs in it. Each blog is routed to a single shard so almost all queries only hit one shard, but we can (and do) search across multiple shards for some use cases.
    • 6 data nodes across 3 DCs running ES 1.3.9. Hosts our WordPress.com VIP indices and lots of other use cases. 321 indices (mostly VIPs). ~8m queries/day. ~1.5m index ops/day. Typical VIP index config is a single shard that is replicated across the three data centers. Most of these indices are small enough that sharding would reduce performance and reduce query relevancy.
    • 12 data nodes across 3 DCs running ES 1.7.5. Primarily powers search.wordpress.com. Indexes the past 6 quarters of all posts. One index per quarter with 30 shards per quarter. Queries typically hit all 180 shards.
    • 3 data nodes across 3 DCs running ES 2.3.1. Currently an experimental cluster as we work to migrate to 2.x. Only production index right now is for en.support.wordpress.com.
    • 15 (and possibly expanding to 100) data nodes for a Logstash cluster running ES 2.3. A lot of logging use cases for many different services. Growing rapidly.
  • All of our clusters use three dedicated master nodes with one master in each data center. The first cluster has its own master nodes. The next three share master servers with multiple instances of ES running on each server.
  • Typical data server config:
    • 96GB RAM with 31GB for ES heap. Remaining gets used for file system caching
    • 1-3 TB of SSD per server. In our testing SSDs are very worthwhile.
  • Query speed:
    • Related Posts: median 44ms; 95th percentile: 190ms; 99th percentile: 650ms. This is way lower than when we launched in 2013 and 99th percentile was 1.7 seconds.
    • VIP Queries: median: 25ms; 95th percentile: 109ms; 99th percentile: 311ms
    • search.wordpress.com queries: median: 130ms; 95th percentile: 250ms; 99th percentile: 260ms
  • Client-side Optimizations:
    • We cache all queries results in memcache which cuts our ES query rate in half
    • memcache timeouts vary from 30 seconds to 36 hours depending on use case
    • We analyze all queries on the client side and optimize the ES filters:
      • have a blacklist of fields that we never cache (blog_id, post_id, author_id) because they have such high cardinality (100m+ unique ids)
      • we rewrite and/or/not filters into bool queries and try to flatten them into a single filter
      • We don’t allow some types of queries (we have a whitelist)
      • We don’t allow facets/aggregations on certain fields (content, title, excerpt)
    • We generally don’t allow paging too deep or returning thousands of results at once
    • A general pattern we use is to use ES to get IDs for content, and then we get the real content from MySQL for displaying to users. This reduces what data ES needs (we strip out HTML), and we can be certain the data is not out of date since ES can be up to 60 seconds out of date in some cases (though typically is less than 5 seconds).
  • Query Use Cases (in order of query frequency):
    • Related Posts
    • Replacing WP_Query calls by converting slow SQL calls to an ES query (WordPress tag/category pages, home pages, etc)
    • search.wordpress.com
    • Language Detection using ES langdetect plugin (used for every post we index)
    • Analyze API (used to perform reliable word counting regardless of language – in conjunction with the langdetect call)
    • Blog Search (replacing the built in WordPress site search)
    • Theme Search
    • Search Queries that are used when reindexing content (eg when a blog’s tag is renamed we need to search for all posts with that tag and reindex them)
    • Various support searches
    • A number of custom VIP use cases
    • A number of custom internal use cases (searching our p2s, suggesting posts that may be relevant to read, searching our internal docs, etc)
    • Calypso /posts and /pages for getting/searching all posts a user has authored across all their blogs (potentially hundreds)
  • ES Plugins Deployed:
    • Whatson for looking at shard distribution, disk usage, index size, etc
    • StatsD for performance monitoring (we also send StatsD data from the client about query speed) See the screenshots of dashboards below.
    • ICU Analysis
    • Langdetect
    • SmartCN and Kuromoji Analyzers
    • Head

 

Since images are always fun, here are our Graphana dashboards for our largest cluster over the past 6 hours. The first is our client-side tracking of query/indexing/etc speed

Screen Shot

Second is our aggregated stats (from the StatsD plugin) about the cluster’s performance:

Screen Shot

This cluster/index has been really solid for us over the past two years since it was last built. We have some known issues that have us stuck on 1.3.4, but we’ve also had times where the cluster went many months without any incidents. In general the incidents we have seen have been caused by external factors (usually over indexing or some other growth in the data).

 

 

 

Open Source Flow Collecting with Elastic, Logstash, and Kibana

Today, most open source network flow tools lack a flexible and easy to use interface. Using Logstash’s built-in netflow codec, Kibana’s great looking and powerful web interface, and the flexibility of Elastic, you can build a tool that rivals commercial flow-collecting products. Continue reading

What’s new in Calypso?

December Edition

It’s been almost a month since we released Calypso and the response has been great from the community. For those following the project more closely, we’ll be publishing summaries on new developments, focusing on framework-level improvements, new components, and the tools contributors have to work with.

Making Calypso more welcoming for developers and designers

If you install Calypso locally and point your browser to calypso.localhost:3000/devdocs/welcome we have a new developer flow that introduces you to our documentation. You can access the docs anytime from the environment badge in the lower left corner, which also highlights the git branch you are in:

Framework

  • Upgraded to Node v4 and React 0.14.
  • Started implementing Redux, a state container solution to manage data flow in the application. If you are interested in this, we are gradually moving our different data modules to Redux.
  • We refined and documented our approach to components.
  • Began exploring how pluggable modules could work in Calypso.
  • Continued migration to use svg icons everywhere instead of the old icon font. We added a few new gridicons as well.

New components and updates

Components are the building blocks of the Calypso UI. We constantly refine them and build new ones, from simple user interface ones to those carrying more complex functionality. This allows us to craft interfaces that are consistent and rich. You can check out all of these if you go to calypso.localhost:3000/devdocs/design, our live components gallery. These are some of the updates we did this month.

Button

We added a borderless variation for one of our most used components.

SelectDropdown

Added a compact variation.

Interval

A utility component, primarily meant for setting up a poller interface wrapping another component.

SitesDropdown

A dropdown component for selecting a site, which includes instant search, handling of private sites.

Cloudup F0CE7CA7 3367 4AC1 88F9 E4CEE8790D4D

FeatureExample

A component that renders other mocked components with a faded effect to illustrate a section when for some reasons the user cannot access it.

Notices

Consolidated Notices into a single component in components/notice. Also added a new compact variation with flex-box magic for narrow layouts:

Site

The core component to display a site-card now support a homeLink prop which turns it into a link to the homepage of the site and renders the following icon on hover:

Draft

Component used to render individual post items. Now supports a “selected” prop to highlight a single draft in a list. (Used in the editor.)

FoldableCard

Now also support custom icons.

Stay tuned to this blog for upcoming Calypso news and updates.

WordPress.com Desktop App Goes Open Source, Linux App Arrives

We are proud to announce the full open sourcing of the WordPress.com desktop app. You can access the source and documentation on GitHub at the automattic/wp-desktop repository.

The core application Calypso was released as open source a few weeks ago, and now the work that went into building the desktop applications using Electron is available as well. Continue reading