Authentication improvements for testing your apps

We’ve just made it easier for developers to authenticate and test API calls with their own applications.

As the client owner, you can now authenticate with the password grant_type, allowing you to skip the authorization step of authenticating, and logging in with your WordPress.com username and password. You can also gain the global scope so that you no longer need to request authentication for each blog you wish to test your code with.

This is especially useful to contributors of the WordPress Android and iOS apps, which previously required special whitelisting on our part.

Here’s an example of how you can get started with using both these features:

Note that if you are using 2-step authentication (highly recommended) you will need to create an application password to be able to use the password grant_type.

$curl = curl_init( "https://public-api.wordpress.com/oauth2/token" );
curl_setopt( $curl, CURLOPT_POST, true );
curl_setopt( $curl, CURLOPT_POSTFIELDS, array(
    'client_id' => your_client_id,
    'client_secret' => your_client_secret_key,
    'grant_type' => 'password'
    'username' => your_wpcom_username,
    'password' => your_wpcom_password,
) );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1);
$auth = curl_exec( $curl );
$auth = json_decode($auth);
$access_key = $auth->access_token;

As noted above, these are only available to you as the owner of the application, and not to any other user. This is meant for testing purposes only.

You can review existing authentication methods here.

If you have any questions, please drop them in the comments or use our contact form to reach us.

REST Development Console — now open source!

For developers working with the REST API, the browser-based API console is an essential debugging tool. It allows you to test your API queries and interactively explore the results (or errors) that the API returns.

REST API console - exploring results

It also puts the documentation at your fingertips and allows you to build a custom query right from any method’s description.

REST API console - building a query

Like the REST API itself, this tool works for any blog on WordPress.com and for any self-hosted WordPress install using Jetpack.

With the addition of implicit OAuth, we’ve released an open-source version of the API console that you can run yourself.

First, you’ll want to create a WordPress.com application (or modify an existing one) and make sure to set the Javascript Origins option. This should be the fully-qualified URL (including http:// or https:// ) of the site you’ll be running the API console on. To run it locally, just use “http://localhost”.

REST API console - JS origins setting

Then, just head on over to the GitHub repository, clone your own copy, and put your application’s info in the file config.json.

Screen Shot 2014-06-10 at 5.23.59 PM

When you’re running the console locally, you can authenticate by clicking on the box in the lower-right corner.

dev console - auth button

Once you’re linked to a blog, the blog’s ID will be shown in the lower right. You can click on it to change which blog you’re working with.

Important note: When you’ve connected to a blog, the console is hooked up to the live database — any changes you make will be reflected on that blog! You might want to create a test blog if you’re planning to make any requests that will modify content.

The REST API console is located at https://developer.wordpress.com/console/.

If you’re using the API, we’d love to hear what you think! Have you used the development console? What’s great (or not so great) about it?

Meet Sulfur — a Media Manager App Built in JavaScript

Since Automattic is a distributed company and a lot of us work from home, we hold meetups to get face-to-face interaction. The whole company meets up once a year and individual teams get together more often. One component of those meetups is a “meetup project” that we all work on together.

The team I lead — “Team I/O*” — just finished a lovely week in Reykjavik, Iceland. Our team is responsible for partnerships and our APIs.

We spent the first day releasing better JavaScript support for our APIs. After that we decided to make an example app, mainly focusing on the new CORS support and implicit OAuth system.

We decided to build a media manager purely in the browser. We picked a codename and Sulfur was born.

Update: Check out the live demo!

Screen Shot 2014-05-20 at 5.18.13 PM

Sulfur is an app built using Backbone, Underscore.js, Plupload, jQuery, MomentJS, Bootstrap, RequireJS, and the WordPress.com JSON REST API.

It shows how you can use Implicit OAuth to do authentication without a server component. It also provides examples for using the API with Backbone and third party libraries like plupload for uploading media.

Sulfur allows you to upload images, view your entire media library contents, view meta data, and delete images. All the code is open source. Check out Sulfur on GitHub.

We learned a lot while building the app and hope to use that knowledge to improve our APIs both for internal and external use.

Screen Shot 2014-05-20 at 9.11.02 PM

Let us know what you think in the comments!

*Named after one of the moons of Jupiter and our love of APIs (input/output)

A brand new Developer Site

As you may have noticed, we’ve just relaunched the WordPress.com Developer site (the very one you’re reading right now!) with a brand new look and feel!

We’ve rebranded the site to match the overall WordPress.com aesthetic as well as to align with the new user management and insights sections we launched just a few weeks ago.

E5A0rwxNNg-3000x3000

The goal of the redesign was not only to modernize the site but make it easier for you, our partners and third-party developers to find the information you are looking for. In addition, we’ve reviewed all of our existing documentation and past blog posts to make sure the information is accurate and relevant.

Over the next few months, you’ll see more updates to the site and more frequent blog posts from our team.

I’d personally like to thank the team that worked on the relaunch with me: Raanan, Kelly, Kat, Justin, and Stephane.

If you’d like to let us know what you think of the new site, report a bug, or have suggestions for future improvements, please comment below, tweet at us @AutomatticEng or contact us privately.

Introducing WordPress.com Insights

We love stats at Automattic. They’re key to understanding our users, and a driving force behind a lot of what we do.

When we make a change, we measure its impact and use the metrics to make data-informed decisions. For example, we recently improved how our Publicized posts look on other services. Using our own data as well as data provided by our partners, we made further improvements and tweaks to increase click throughs to your posts.

Our partners and those building on our platform should have the same ability. That’s why we spent the past few weeks creating tool to support them: WordPress.com Insights.

Check it out in the short video below:

Music Credit: Anthony Vitale

Insights provides data and graphs for a variety of metrics: Connections/authorizations, API calls, API errors, posts published, WordPress.com Connect Logins, and the reach of posts published from your app. By exposing this data, developers can measure the impact of their integrations over time.

In addition to graphs, we’ve also built in export functionality allowing you to get your data in CSV format. Since Insights is built using the REST API, you can pull data out in JSON format as well.

To accompany Insights, we also added the ability to allow other users and developers access to your app. To edit your permissions, head to the Apps Manager, select an application, and click “manage users.” If you want to provide temporary access you can also generate sharable URLs.

For more information and a walkthrough of Insights, check out our documentation.

An efficient alternative to paging with SQL OFFSETs

Challenge

Running WordPress.com means having multimillion-record database tables. Tables which we often need to batch-query.

Provided we could hardly select (or update, etc) millions of records at once and expect speed, we commonly have to “page” our scripts to only handle a limited number of records at once, then move on to the next batch.

Classic, but inefficient, solution

The usual way of paging result sets in most SQL RDMS is to use the OFFSET option (or LIMIT [offset], [limit], which is the same).

SELECT * FROM my_table OFFSET 8000000 LIMIT 100;

But on a performance level, this means you’re asking your DB engine to figure out where to start from all on its own, every time. Which then means it must be aware of every record before the queried offset, because they could be different between queries (deletes, etc). So the higher your offset number, the longer the overall query will take.

Alternative solution

Instead, of keeping track of an offset in your query script, consider keeping track of the last record’s primary key in the previous result set instead. Say, its ID. At the next loop instance, query your table based on other records having a greater value for said ID.

SELECT * FROM my_table WHERE id > 7999999 LIMIT 100;

This will let you page in the same way, but your DB’s engine will know exactly where to start, based on an efficient indexed key, and won’t have to consider any of the records prior to your range. Which will all translate to speedy queries.

Here’s a real-life sample of how much difference this can make:

mysql> SELECT * FROM feeds LIMIT 8000000, 10;
  [...]
10 rows in set (12.80 sec)

mysql> SELECT * FROM feeds WHERE feed_id > 12958559 LIMIT 10;
  [...]
10 rows in set (0.01 sec)

I received the very same records back, but the first query took 12.80 seconds, while the alternative took 0.01 instead. :)

PHP/WordPress example

<?php
// Start with 0
$last_id = 0;

do {
    $blogs = $wpdb->get_results( $wpdb->prepare(
        'SELECT * FROM wp_blogs WHERE blog_id > %d LIMIT 100;',
        $last_id // Use the last ID to start after
    ) );

    foreach ( $blogs as $blog ) {
        // Do your thing!
        // ...
        // Record the last ID for the next loop
        $last_id = $blog->blog_id;
    }
// Do it until we have no more records
} while ( ! empty( $blogs ) );
?>

Like elasticsearch? We do too!

Elasticsearch tools

Elasticsearch, if you’re not familiar with it, is defined as a distributed restful search and analytics tool.

When it comes to implementing such an infrastructure, our developers not only face the challenges involved in indexing tens of millions of sites with grace and skill, they also write quite extensively about their related adventures, so others can benefit from their experiences.

You can find a plethora of posts on Greg Brown’s blog, under the appropriate tag. Subjects ranging from performance and scaling, all the way to “Elasticsearch, Open Source, and the Future“. And in true Automattician fashion, he isn’t even shy about recognizing his mistakes.

But Greg is not alone! Xiao Yu also recently wrote about the tools he uses, and a plugin he concocted for his own needs:

I’ve taken all that I wished I could do with both of those plugins and created a new Elasticsearch plugin that I call Whatson. This plugin utilizes the power of D3.js to visualize the nodes, indices, and shards within a cluster. It also allows the drilling down to segment data per index or shard. With the focus on visualizing large clusters and highlighting potential problems within. I hope this plugin helps others find and diagnose issues so give it a try.

How’s that for advanced? :)