Cloud-hosted Branches for Painless Testing

Anyone familiar with developing web applications knows how tedious it is to test code changes made by your peers. Whether it’s a designer learning how to build and run a local server, or even a developer that needs to stash local changes and wait for a server rebuild, testing changes is a major workflow interruption that can mean minutes of thumb twiddling. At WordPress.com, we’re proud that we’ve made testing in-progress work a quick process that’s as easy as sharing a url.

Introducing Calypso.live

If you’ve ever seen a Calypso pull request, you might have noticed that our helpful matticbot always comments on the request with a link to calypso.live. If you follow the link, it takes you to a hosted version of that branch. If you’re the first person to access the branch, you may have to wait a couple of minutes for the app to be built, during which you’ll be greeted with this loading screen:

calypso.live loading screen

Once the server is done building, it presents you with a version of Calypso corresponding to that branch of code. This makes it easy and quick for developers to review functionality and designers to review the look and feel, all without having to manage local development servers.

One piece of good news is that you usually aren’t the first to access a branch. That’s because our automated tests actually rely on these cloud-hosted branches to test against.

How does this work technically?

Calypso.live is powered by a specialized docker server named dserve, which was written to work for any Docker-based web application. It automatically manages thousands of versions of an app and makes both each git sha and branch accessible by url. An example request might go like this:

1. Request is made for calypso.live?branch=my-branch
2. Once dserve receives the request, it checks to see if my-branch is ready to be served
3. Assuming a container is already running for my-branch, dserve proxies the request to the right container — very similar to how a load balancer sits in front of an array of servers and proxies the request.

Get involved!

Feel free to check out the dserve repository, our e2e tests repository, or wp-calypso, make a fork, and provide us with any feedback or suggestions. Pull requests are always welcome.

***

Jake Fried is a Software Engineer at Automattic, mainly working on Calypso.

How Canaries Help Us Merge Good Pull Requests

At WordPress.com we strive to provide a consistent and reliable user experience as we merge and release hundreds of code changes each week.

We run automated unit and component tests for our Calypso user interface on every commit against every pull request (PR).

We also have 32 automated end-to-end (e2e) test scenarios that, until recently, we would only automatically run across our platform after merging and deploying to production. While these e2e scenarios have found regressions fairly quickly after deploying (the 32 scenarios execute in parallel in just 10 minutes), they don’t prevent us from merging and releasing regressions to our customer experience.

Introducing our Canaries

Earlier this year we decided to identify three of our 32 automated end-to-end test scenarios that would act as our “canaries”: a minimal subset of automated tests to quickly tell us if our most important flows are broken. These tests execute after a pull request is merged and deployed to our staging environment, but before we deploy the changes to all our customers in production.

These canaries have been very successful in preventing us from deploying regressions to production, however, running these after merging to master (and automatically deploying code to staging) means we’d have to revert code changes if something was wrong. This wasn’t good enough.

Last month we took our canaries to the next level. Instead of just running canaries on merging to master, we now execute canaries against live pull requests and provide feedback to the pull request itself about the canary test status.

How does it work?

Our process is that if you’re a developer working on a pull request for Calypso and it’s ready to review, you add the “[Status] Needs Review” label to alert someone to review your code. Adding this label automatically triggers the e2e canary tests against your pull request:

The results are separate from the unit and component tests which already run against every pull request (on every push).

How does this technically work?

Our automated e2e tests are open-source, but they reside separately from our Calypso GitHub code repository. This is because the e2e scenarios represent the entire WordPress.com customer experience: they’re not just automated Calypso user interface tests. For example, our tests include verifying that our customers receive appropriate emails that are not part of the Calypso code base.

We “connect” our two projects using CircleCI builds and a custom “bridge” written in Node.js (which is also open-source). This bridge provides webhooks for GitHub pull requests to execute CircleCI builds using the CircleCI API. It reports the status of these builds using the GitHub status API. We do apply a little bit of cleverness in that we can match branch names so we can make changes to our e2e tests that correspond to changes to our Calypso changes. Our bridge runs on Automattic’s VIP Go platform.

A summary and what’s next?

Running our canaries on pull requests has been a great success. Developers love the confidence the canaries give them in knowing that our key end-to-end scenarios won’t regress when introducing changes rapidly.

We’d now like to expand the bridge’s scope to optionally run the full set of 32 end-to-end automated tests on pull requests that have a broader impact, changes like upgrading a dependency or refactoring a framework design pattern. This again will give our developers even greater confidence in the ability to merge code and provide a consistent and reliable experience to our customers.

Get involved!

Feel free to check out our e2e tests repository, or our bridge repository, make a fork, and provide us with any feedback or suggestions. Pull requests are always welcomed.

***

Alister Scott is an Excellence Wrangler for Automattic and blogs regularly about software testing at his blog WatirMelon.

New WordPress unified API console

Since the WordPress 4.7 “Vaughan” release, each WordPress installation includes REST API endpoints to access and manipulate its content.  These endpoints will be the foundation for the next generation of WordPress websites and applications.

Today we’re releasing a brand new Open Source WordPress API console. You can use it to try these endpoints and explore the results.  The console works for any website on WordPress.com and also for any self-hosted WordPress installation.

Using the console with WordPress.com APIs

You can use this application today to make read and write requests to the WordPress.com API or the WordPress REST API for any website hosted on WordPress.com or using Jetpack.  Visit the new version of the application here:  https://developer.wordpress.com/docs/api/console/

Using the console with your self-hosted WordPress sites

To use the console with your self-hosted WordPress installation(s), you’ll need to download the application from GitHub, configure it, and run it on your local machine.  You’ll also need to install the WP REST API – OAuth 1.0a Server plugin on your WordPress site.  The Application Passwords plugin is another option – but – if you use this plugin, make sure that your site is running over HTTPS.  Otherwise, this configuration is insecure.

Full installation and configuration instructions are on the GitHub repository.

Technical Details

The console is a React/Redux application based on create-react-app that persists its state to localStorage.

What’s next?

We have a few more features planned that we think you’ll like.

• We can use the new console application to allow you to easily generate and save OAuth2 tokens for your WordPress.com API Applications.  As compared to implementing the OAuth2 flow yourself, this will be a much easier way to obtain an API token for testing your applications.
• We also plan to ship the console as a regular WordPress plugin, replacing the existing older plugin.
• We can allow you to add/edit self-hosted WordPress websites on our hosted version of the console and persist them to localStorage.  This way you’ll be able to query your WordPress sites without having to install the console yourself.

Contribute

As usual, the new console is open source, and we hope this will be a tool that will benefit the entire WordPress community.

If you find a bug, think of a new feature, or want to make some modifications to the API console, feel free to look through existing issues and open a new issue or a PR on the GitHub repository.  We welcome all contributions.

Get.blog Goes Open Source

We’re happy to announce that we’ve open-sourced two projects last week: Delphin and its sibling, Delphin Bootstrap. Delphin powers get.blog, a site we built for a smooth registration of .blog domains. Delphin Bootstrap is a portable development environment for Delphin — it makes getting Delphin up and running easy on any operating system.

One of Delphin’s main goals was to simplify the process of registering and managing a domain. We’ve focused on the user experience, trying to avoid as much industry jargon as possible, while keeping in mind that purchasing a domain is just the first step in the journey toward a larger goal, like telling a story or selling products.

Delphin is a web application that uses parts of the WordPress.com API, as well as a few more specific endpoints built on the new WordPress REST API. Like Calypso — the WordPress.com front end — it is based on technologies such as React, Redux, and Webpack. We also used Delphin as a platform to experiment with new technologies including Yarn, Jest, CSS modules, and React Router. Some of these experiments will make it back to Calypso.

We wanted to share some of our findings:

• Yarn is a fantastic package manager. We had a few hiccups when we integrated it with CircleCI, but it now works smoothly.
• CSS modules was a great addition to this project, as it makes it possible to use semantic names for the CSS classes of components, instead of coming up with a convention to prevent namespace collisions in a global stylesheet.
• We ended up using a lot of wonderful tools and modules shared by the community.

At Automattic, we’ve been open-sourcing code for more than a decade now, and have more than 350 repositories on GitHub. We’re excited to add Delphin to this list, and welcome contributions, feedback, or reported issues!

WordPress REST API on WordPress.com

As you might know if you follow WordPress developer news, the WordPress REST API content endpoints are slated to be merged into WordPress core in the upcoming 4.7 release. These endpoints cover basic WordPress features (posts, pages, custom post types, revisions, media, comments, taxonomies, users, and settings). While they have also been available via the WP REST API plugin for some time, this is the latest step in a multi-year project to bring a modern, RESTful API to all WordPress installations.

We are pleased to announce that WordPress.com is now running the WordPress REST infrastructure alongside our pre-existing v1 API. Additionally, we have made the new content endpoints available now so that developers can try them out, provide feedback and drive wider adoption of the new API.

The success metrics for this feature, detailed here, focus on increasing utilization of the new API in plugins, themes, third-party clients and within WordPress core itself. This new API represents the future of WordPress and WordPress.com, and we want to make it as solid as we can. We encourage all WordPress developers to take it for a spin and let us know what you think!

Using the New API

The WordPress REST API is available on WordPress.com with the following base URL: https://public-api.wordpress.com/wp/v2/. This new API is backwards-incompatible with the v1 API, which we have no plans to deprecate at this time.

Going forward, we encourage developers to write new code against the v2 API whenever possible. This has the added benefit of making your applications able to run against both WordPress.com and self-hosted WordPress sites.

Documentation for the WordPress REST API and its endpoints can be found at http://v2.wp-api.org/.  Our installation of the API uses the same OAuth2 authentication tokens as our version 1 API.

To make testing and exploration easier, we’ve also updated our API console to support both versions of our API. You can switch API versions using the dropdown toggle in the upper left:

 

Once you’ve selected the WP REST API, you can view and select endpoints and make authenticated requests just as in the previous version:

Caveats

For self-hosted sites running the Jetpack plugin, we expect read requests to work (but you must be running at least Jetpack version 4.3.2).  Write requests to Jetpack sites do not work yet, since this will require the site to be running the content endpoints and a future version of the Jetpack plugin that supports them.

In addition, a few endpoints are restricted or disabled on WordPress.com, generally for parity with our v1 API. A few examples:

• Listing and retrieving media items requires authentication
• Listing and retrieving user info requires authentication
• Endpoints to create/edit/delete users are disabled
• Updating the email and URL settings via the settings endpoint is disabled

Examples

Here are some examples that could serve as a starting point. If you build your own, please share it in the comments!

• Our REST API console: https://github.com/Automattic/rest-api-console2 (supports WordPress.com sites)
• The in-progress rewrite of our REST API console: https://github.com/Automattic/wp-api-console (supports both WordPress.com and self-hosted sites)
• A simple “secret notepad” backed by the REST API (supports both WordPress.com and self-hosted sites): https://github.com/nylen/wp-rest-notepad
• A simple command-line todo list app backed by the REST API (supports WordPress.com sites): https://github.com/timmyc/wptodo
• A proof-of-concept of Quick Draft using the REST API (for self-hosted sites): https://core.trac.wordpress.org/ticket/38342
• A proof-of-concept of Press This using the REST API (for self-hosted sites): https://core.trac.wordpress.org/ticket/38343
• We’re also using custom API endpoints built on the REST infrastructure in several places, for example in our Jetpack and WooCommerce plugins and on WordPress.com.  Since we enabled them in early 2016, our custom WordPress.com endpoints have served over 200 million requests.

How to Provide Feedback

• There are weekly developer chats in the WordPress Slack channel. More info can be found on the sidebar of https://make.wordpress.org/core/.
• WordPress core uses Trac to track bugs and open issues. The master ticket for this feature is here: https://core.trac.wordpress.org/ticket/38373; you can also search for an existing REST API issue or open a new one.
• Drop a comment on this post, or contact our developer support team, especially if you find a problem or inconsistency specific to the WordPress.com version of the API.

Improved REST API Support for Custom Taxonomies

The WordPress.com REST API has always enabled developers to manage categories and tags for the posts on a site. Until now, however, if a site had enabled any custom post types with custom taxonomies, developers were out of luck if they had hoped to manage custom terms using the REST API.

We’re happy to announce that this is no longer the case, with improved support for custom taxonomies. This includes a handful of new endpoints and enhancements to existing endpoints.

Specifically, you can now…

• Retrieve a list of terms for a site
• Retrieve a single term for a site
• Create a new term
• Update an existing term
• Delete a term
• List taxonomies associated with a post type
• Retrieve a post including terms for all public taxonomies
• Create or update a post including terms for any taxonomy
• List posts filtered by a custom taxonomy term (available now for WordPress.com sites and in a future Jetpack release)

Note that taxonomies and terms are only included in a response if the taxonomy is registered as public or if the authorization token sent with the request has the proper capabilities for viewing or managing that taxonomy. Refer to the register_taxonomy function documentation for more information.

As an example, if a Jetpack site supported a custom book post type with a genre taxonomy, I could create a new book by issuing the following cURL command:

curl \
 -H 'authorization: Bearer YOUR_API_TOKEN' \
 --data-urlencode 'title=Sample Book' \
 --data-urlencode 'type=book' \
 --data-urlencode 'terms[genre][]=Fiction' \
 'https://public-api.wordpress.com/rest/v1.2/sites/example.com/posts/new/'

If you’re new to working with the WordPress.com REST API, we recommend you head on over to our Getting Started with the API guide to learn more.

Introducing our WordPress.com Automated e2e Tests

At WordPress.com we strive to continuously deliver new functionality whilst ensuring that we provide a consistent and reliable user experience.

One key element to all of this is developing self-testing code; that is, creating and maintaining a comprehensive suite of automated tests that are developed in conjunction with our functional code.

The WordPress Calypso User Interface supports React unit and component testing. Because the WordPress.com user experience depends on many other factors beyond just the precision of our Calypso components, we have also been building a suite of automated end-to-end (e2e) tests that verify our critical user flows are working as intended.

Now, the WordPress.com team is proud to announce that we have released our automated e2e test suite as open-source. You can find it here on GitHub.

Why open source is so important

You could say our automated testing pyramid looks like this:

The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.

~ Matt Mullenweg, Automattic CEO

The technical details

Our tests use the ES2015 syntax on Node, which should make it it easy for anyone familiar with Calypso to contribute.

We are utilizing the open-source selenium-webdriver project to drive real user browsers. We write our e2e tests in the same test specifications library as Calypso (Mocha) and we run these on the same CI platform that being on headless CircleCI Linux containers.

Since our platform is fully responsive, we run the same tests (with slightly different logic) against three different screen resolutions to ensure our customer experience functions across different device types.

We’ve also done some neat things like developing real-time Slack notifications (with screenshots) when tests are failing so we can investigate quickly.

It’s still early days, and we’re still striving to make these tests as deterministic and reliable as possible.

You can run these tests against a local WordPress Calypso development environment. We automatically run these tests on every WordPress.com deployment, as well as every six hours, to ensure end-to-end flows are working as intended for our users. We are planning to run these, or a subset of these, against pull requests as they are developed to prevent bugs being introduced into our platform.

Automated tests have already been successful in making us aware of issues sooner so we can fix them faster.

Get involved!

Feel free to check out our repository, make a fork, and provide us with feedback or suggestions. Pull requests are always welcomed.

***

Alister Scott is an Excellence Wrangler for Automattic and blogs regularly about software testing at WatirMelon.