New WordPress unified API console

Since the WordPress 4.7 “Vaughan” release, each WordPress installation includes REST API endpoints to access and manipulate its content.  These endpoints will be the foundation for the next generation of WordPress websites and applications.

Today we’re releasing a brand new Open Source WordPress API console. You can use it to try these endpoints and explore the results.  The console works for any website on WordPress.com and also for any self-hosted WordPress installation.

Using the console with WordPress.com APIs

You can use this application today to make read and write requests to the WordPress.com API or the WordPress REST API for any website hosted on WordPress.com or using Jetpack.  Visit the new version of the application here:  https://developer.wordpress.com/docs/api/console/

Using the console with your self-hosted WordPress sites

To use the console with your self-hosted WordPress installation(s), you’ll need to download the application from GitHub, configure it, and run it on your local machine.  You’ll also need to install the WP REST API – OAuth 1.0a Server plugin on your WordPress site.  The Application Passwords plugin is another option – but – if you use this plugin, make sure that your site is running over HTTPS.  Otherwise, this configuration is insecure.

Full installation and configuration instructions are on the GitHub repository.

Technical Details

The console is a React/Redux application based on create-react-app that persists its state to localStorage.

What’s next?

We have a few more features planned that we think you’ll like.

• We can use the new console application to allow you to easily generate and save OAuth2 tokens for your WordPress.com API Applications.  As compared to implementing the OAuth2 flow yourself, this will be a much easier way to obtain an API token for testing your applications.
• We also plan to ship the console as a regular WordPress plugin, replacing the existing older plugin.
• We can allow you to add/edit self-hosted WordPress websites on our hosted version of the console and persist them to localStorage.  This way you’ll be able to query your WordPress sites without having to install the console yourself.

Contribute

As usual, the new console is open source, and we hope this will be a tool that will benefit the entire WordPress community.

If you find a bug, think of a new feature, or want to make some modifications to the API console, feel free to look through existing issues and open a new issue or a PR on the GitHub repository.  We welcome all contributions.

Get.blog Goes Open Source

We’re happy to announce that we’ve open-sourced two projects last week: Delphin and its sibling, Delphin Bootstrap. Delphin powers get.blog, a site we built for a smooth registration of .blog domains. Delphin Bootstrap is a portable development environment for Delphin — it makes getting Delphin up and running easy on any operating system.

One of Delphin’s main goals was to simplify the process of registering and managing a domain. We’ve focused on the user experience, trying to avoid as much industry jargon as possible, while keeping in mind that purchasing a domain is just the first step in the journey toward a larger goal, like telling a story or selling products.

Delphin is a web application that uses parts of the WordPress.com API, as well as a few more specific endpoints built on the new WordPress REST API. Like Calypso — the WordPress.com front end — it is based on technologies such as React, Redux, and Webpack. We also used Delphin as a platform to experiment with new technologies including Yarn, Jest, CSS modules, and React Router. Some of these experiments will make it back to Calypso.

We wanted to share some of our findings:

• Yarn is a fantastic package manager. We had a few hiccups when we integrated it with CircleCI, but it now works smoothly.
• CSS modules was a great addition to this project, as it makes it possible to use semantic names for the CSS classes of components, instead of coming up with a convention to prevent namespace collisions in a global stylesheet.
• We ended up using a lot of wonderful tools and modules shared by the community.

At Automattic, we’ve been open-sourcing code for more than a decade now, and have more than 350 repositories on GitHub. We’re excited to add Delphin to this list, and welcome contributions, feedback, or reported issues!

WordPress REST API on WordPress.com

As you might know if you follow WordPress developer news, the WordPress REST API content endpoints are slated to be merged into WordPress core in the upcoming 4.7 release. These endpoints cover basic WordPress features (posts, pages, custom post types, revisions, media, comments, taxonomies, users, and settings). While they have also been available via the WP REST API plugin for some time, this is the latest step in a multi-year project to bring a modern, RESTful API to all WordPress installations.

We are pleased to announce that WordPress.com is now running the WordPress REST infrastructure alongside our pre-existing v1 API. Additionally, we have made the new content endpoints available now so that developers can try them out, provide feedback and drive wider adoption of the new API.

The success metrics for this feature, detailed here, focus on increasing utilization of the new API in plugins, themes, third-party clients and within WordPress core itself. This new API represents the future of WordPress and WordPress.com, and we want to make it as solid as we can. We encourage all WordPress developers to take it for a spin and let us know what you think!

Using the New API

The WordPress REST API is available on WordPress.com with the following base URL: https://public-api.wordpress.com/wp/v2/. This new API is backwards-incompatible with the v1 API, which we have no plans to deprecate at this time.

Going forward, we encourage developers to write new code against the v2 API whenever possible. This has the added benefit of making your applications able to run against both WordPress.com and self-hosted WordPress sites.

Documentation for the WordPress REST API and its endpoints can be found at http://v2.wp-api.org/.  Our installation of the API uses the same OAuth2 authentication tokens as our version 1 API.

To make testing and exploration easier, we’ve also updated our API console to support both versions of our API. You can switch API versions using the dropdown toggle in the upper left:

 

Once you’ve selected the WP REST API, you can view and select endpoints and make authenticated requests just as in the previous version:

Caveats

For self-hosted sites running the Jetpack plugin, we expect read requests to work (but you must be running at least Jetpack version 4.3.2).  Write requests to Jetpack sites do not work yet, since this will require the site to be running the content endpoints and a future version of the Jetpack plugin that supports them.

In addition, a few endpoints are restricted or disabled on WordPress.com, generally for parity with our v1 API. A few examples:

• Listing and retrieving media items requires authentication
• Listing and retrieving user info requires authentication
• Endpoints to create/edit/delete users are disabled
• Updating the email and URL settings via the settings endpoint is disabled

Examples

Here are some examples that could serve as a starting point. If you build your own, please share it in the comments!

• Our REST API console: https://github.com/Automattic/rest-api-console2 (supports WordPress.com sites)
• The in-progress rewrite of our REST API console: https://github.com/Automattic/wp-api-console (supports both WordPress.com and self-hosted sites)
• A simple “secret notepad” backed by the REST API (supports both WordPress.com and self-hosted sites): https://github.com/nylen/wp-rest-notepad
• A simple command-line todo list app backed by the REST API (supports WordPress.com sites): https://github.com/timmyc/wptodo
• A proof-of-concept of Quick Draft using the REST API (for self-hosted sites): https://core.trac.wordpress.org/ticket/38342
• A proof-of-concept of Press This using the REST API (for self-hosted sites): https://core.trac.wordpress.org/ticket/38343
• We’re also using custom API endpoints built on the REST infrastructure in several places, for example in our Jetpack and WooCommerce plugins and on WordPress.com.  Since we enabled them in early 2016, our custom WordPress.com endpoints have served over 200 million requests.

How to Provide Feedback

• There are weekly developer chats in the WordPress Slack channel. More info can be found on the sidebar of https://make.wordpress.org/core/.
• WordPress core uses Trac to track bugs and open issues. The master ticket for this feature is here: https://core.trac.wordpress.org/ticket/38373; you can also search for an existing REST API issue or open a new one.
• Drop a comment on this post, or contact our developer support team, especially if you find a problem or inconsistency specific to the WordPress.com version of the API.

Improved REST API Support for Custom Taxonomies

The WordPress.com REST API has always enabled developers to manage categories and tags for the posts on a site. Until now, however, if a site had enabled any custom post types with custom taxonomies, developers were out of luck if they had hoped to manage custom terms using the REST API.

We’re happy to announce that this is no longer the case, with improved support for custom taxonomies. This includes a handful of new endpoints and enhancements to existing endpoints.

Specifically, you can now…

• Retrieve a list of terms for a site
• Retrieve a single term for a site
• Create a new term
• Update an existing term
• Delete a term
• List taxonomies associated with a post type
• Retrieve a post including terms for all public taxonomies
• Create or update a post including terms for any taxonomy
• List posts filtered by a custom taxonomy term (available now for WordPress.com sites and in a future Jetpack release)

Note that taxonomies and terms are only included in a response if the taxonomy is registered as public or if the authorization token sent with the request has the proper capabilities for viewing or managing that taxonomy. Refer to the register_taxonomy function documentation for more information.

As an example, if a Jetpack site supported a custom book post type with a genre taxonomy, I could create a new book by issuing the following cURL command:

curl \
 -H 'authorization: Bearer YOUR_API_TOKEN' \
 --data-urlencode 'title=Sample Book' \
 --data-urlencode 'type=book' \
 --data-urlencode 'terms[genre][]=Fiction' \
 'https://public-api.wordpress.com/rest/v1.2/sites/example.com/posts/new/'

If you’re new to working with the WordPress.com REST API, we recommend you head on over to our Getting Started with the API guide to learn more.

Introducing our WordPress.com Automated e2e Tests

At WordPress.com we strive to continuously deliver new functionality whilst ensuring that we provide a consistent and reliable user experience.

One key element to all of this is developing self-testing code; that is, creating and maintaining a comprehensive suite of automated tests that are developed in conjunction with our functional code.

The WordPress Calypso User Interface supports React unit and component testing. Because the WordPress.com user experience depends on many other factors beyond just the precision of our Calypso components, we have also been building a suite of automated end-to-end (e2e) tests that verify our critical user flows are working as intended.

Now, the WordPress.com team is proud to announce that we have released our automated e2e test suite as open-source. You can find it here on GitHub.

Why open source is so important

You could say our automated testing pyramid looks like this:

The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.

~ Matt Mullenweg, Automattic CEO

The technical details

Our tests use the ES2015 syntax on Node, which should make it it easy for anyone familiar with Calypso to contribute.

We are utilizing the open-source selenium-webdriver project to drive real user browsers. We write our e2e tests in the same test specifications library as Calypso (Mocha) and we run these on the same CI platform that being on headless CircleCI Linux containers.

Since our platform is fully responsive, we run the same tests (with slightly different logic) against three different screen resolutions to ensure our customer experience functions across different device types.

We’ve also done some neat things like developing real-time Slack notifications (with screenshots) when tests are failing so we can investigate quickly.

It’s still early days, and we’re still striving to make these tests as deterministic and reliable as possible.

You can run these tests against a local WordPress Calypso development environment. We automatically run these tests on every WordPress.com deployment, as well as every six hours, to ensure end-to-end flows are working as intended for our users. We are planning to run these, or a subset of these, against pull requests as they are developed to prevent bugs being introduced into our platform.

Automated tests have already been successful in making us aware of issues sooner so we can fix them faster.

Get involved!

Feel free to check out our repository, make a fork, and provide us with feedback or suggestions. Pull requests are always welcomed.

***

Alister Scott is an Excellence Wrangler for Automattic and blogs regularly about software testing at WatirMelon.

State of WordPress.com Elasticsearch Systems 2016

We get asked periodically about how extensively we are using Elasticsearch. And it has come up twice in the past week, so time to write a blog post.

We are constantly expanding what we are using Elasticsearch for and so although some previous posts have broadly define what we are doing, they don’t really capture the continually expanding scale.

So here are some quick bullet points about what we currently have deployed:

• Five clusters with a mix of versions:
  • 42 data nodes spread across 3 US data centers running ES 1.3.4. This cluster mostly runs related posts queries. 1925 shards. 11B docs. 43TB of data. 60m queries/day. 12m index ops/day (has been as high as 940m in a day though). Each index is 175 shards and has 10m blogs in it. Each blog is routed to a single shard so almost all queries only hit one shard, but we can (and do) search across multiple shards for some use cases.
  • 6 data nodes across 3 DCs running ES 1.3.9. Hosts our WordPress.com VIP indices and lots of other use cases. 321 indices (mostly VIPs). ~8m queries/day. ~1.5m index ops/day. Typical VIP index config is a single shard that is replicated across the three data centers. Most of these indices are small enough that sharding would reduce performance and reduce query relevancy.
  • 12 data nodes across 3 DCs running ES 1.7.5. Primarily powers search.wordpress.com. Indexes the past 6 quarters of all posts. One index per quarter with 30 shards per quarter. Queries typically hit all 180 shards.
  • 3 data nodes across 3 DCs running ES 2.3.1. Currently an experimental cluster as we work to migrate to 2.x. Only production index right now is for en.support.wordpress.com.
  • 15 (and possibly expanding to 100) data nodes for a Logstash cluster running ES 2.3. A lot of logging use cases for many different services. Growing rapidly.
• All of our clusters use three dedicated master nodes with one master in each data center. The first cluster has its own master nodes. The next three share master servers with multiple instances of ES running on each server.
• Typical data server config:
  • 96GB RAM with 31GB for ES heap. Remaining gets used for file system caching
  • 1-3 TB of SSD per server. In our testing SSDs are very worthwhile.
• Query speed:
  • Related Posts: median 44ms; 95th percentile: 190ms; 99th percentile: 650ms. This is way lower than when we launched in 2013 and 99th percentile was 1.7 seconds.
  • VIP Queries: median: 25ms; 95th percentile: 109ms; 99th percentile: 311ms
  • search.wordpress.com queries: median: 130ms; 95th percentile: 250ms; 99th percentile: 260ms
• Client-side Optimizations:
  • We cache all queries results in memcache which cuts our ES query rate in half
  • memcache timeouts vary from 30 seconds to 36 hours depending on use case
  • We analyze all queries on the client side and optimize the ES filters:
    • have a blacklist of fields that we never cache (blog_id, post_id, author_id) because they have such high cardinality (100m+ unique ids)
    • we rewrite and/or/not filters into bool queries and try to flatten them into a single filter
    • We don’t allow some types of queries (we have a whitelist)
    • We don’t allow facets/aggregations on certain fields (content, title, excerpt)
  • We generally don’t allow paging too deep or returning thousands of results at once
  • A general pattern we use is to use ES to get IDs for content, and then we get the real content from MySQL for displaying to users. This reduces what data ES needs (we strip out HTML), and we can be certain the data is not out of date since ES can be up to 60 seconds out of date in some cases (though typically is less than 5 seconds).
• Query Use Cases (in order of query frequency):
  • Related Posts
  • Replacing WP_Query calls by converting slow SQL calls to an ES query (WordPress tag/category pages, home pages, etc)
  • search.wordpress.com
  • Language Detection using ES langdetect plugin (used for every post we index)
  • Analyze API (used to perform reliable word counting regardless of language – in conjunction with the langdetect call)
  • Blog Search (replacing the built in WordPress site search)
  • Theme Search
  • Search Queries that are used when reindexing content (eg when a blog’s tag is renamed we need to search for all posts with that tag and reindex them)
  • Various support searches
  • A number of custom VIP use cases
  • A number of custom internal use cases (searching our p2s, suggesting posts that may be relevant to read, searching our internal docs, etc)
  • Calypso /posts and /pages for getting/searching all posts a user has authored across all their blogs (potentially hundreds)
• ES Plugins Deployed:
  • Whatson for looking at shard distribution, disk usage, index size, etc
  • StatsD for performance monitoring (we also send StatsD data from the client about query speed) See the screenshots of dashboards below.
  • ICU Analysis
  • Langdetect
  • SmartCN and Kuromoji Analyzers
  • Head

 

Since images are always fun, here are our Graphana dashboards for our largest cluster over the past 6 hours. The first is our client-side tracking of query/indexing/etc speed

Second is our aggregated stats (from the StatsD plugin) about the cluster’s performance:

This cluster/index has been really solid for us over the past two years since it was last built. We have some known issues that have us stuck on 1.3.4, but we’ve also had times where the cluster went many months without any incidents. In general the incidents we have seen have been caused by external factors (usually over indexing or some other growth in the data).

 

 

 

Open Source Flow Collecting with Elastic, Logstash, and Kibana

Today, most open source network flow tools lack a flexible and easy to use interface. Using Logstash’s built-in netflow codec, Kibana’s great looking and powerful web interface, and the flexibility of Elastic, you can build a tool that rivals commercial flow-collecting products.