Rewriting WordPress’ juju charms for security and HA on OpenStack

Canonical’s IS department is responsible for running most of the company’s external and internal services. This includes services like Landscape and Launchpad as well as 3rd party software including internal and external wikis, WordPress blogs and Django sites.

All new services are deployed with Juju and Mojo onto our OpenStack clouds but we have some legacy services running on bare metal that we have yet to migrate. One of them was WordPress. The main requirements we had for running it in production were:

• allow horizontal scaling
• ensure there are no SPOFs (single points of failure) in the deployment
• lock down filesystem permissions. We explicitly did not want to allow WordPress code to be able to modify itself or to be able to write to any place on disk other than wp-content/uploads/ directory
• use of apparmor Mandatory Access Control (MAC) to limit WordPress and PHP 0-day vulnerabilities impact
• it should work in an environment with strict ingress and egress firewall rules, mostly with minimal Internet access
• support an easy way to upgrade WordPress code using charms without unrestricted access to the Internet
• support plugin and theme installation and updates using charms. It should work in environments with limited Internet access
• support caching by relating with existing charms i.e. squid or varnish
• ensure WordPress admin traffic is encrypted

We decided that the best approach was to write new charms using latest tools available to charm authors. Most notably we now have “charm-helpers” (lp:charm-helpers) which bundles most commonly used python functions in one place.

One of the recent additions to the charm-helpers is a “services” framework. It allows charm authors to focus on describing desired state of the system rather than thinking about hooks and scripts running within them.

The services framework is by far the best way we have come across to write charms so we decided to use it for our charm.

When working on the project we realised that web server support does not really belong in the WordPress charm itself. We wanted to make the design modular and allow people to replace components, for example to replace Apache with Nginx. This led to the decision to split the web server into its own subordinate charm.

Next on the list was plugin and theme support. Those looked like good candidates to split out of the main codebase into their own subordinate charms.

The final piece to work out was horizontal scaling for WordPress. Most of the content is in a database so we did not have to worry about posts, comments or user accounts. There was, however, a small bit that we had to keep in sync between all servers running WordPress code for a given blog: media files. In normal deployments the files end up in wp-content/uploads/.

We needed a way to store those objects in a place that would be resilient, accessible from all WordPress instances, and easy to deploy on OpenStack. All we had to do is connect the dots to come up with OpenStack object storage (Swift) as the solution. It’s available in all our clouds, and it has been around for a while so it’s very stable, well documented, and widely used.

There were some downsides to this solution. WordPress does not support it natively and we could find no existing plugins. So we decided to write our own, or more specifically fork the IBM bluemix support plugin. Our work is available in Launchpad:
lp:~canonical-sysadmins/wordpress/wp-plugin-swift-storage

Another downside is that horizontal scaling only works in clouds with object storage supported by WordPress plugins. Currently this limits us to deploying onto OpenStack. We have not tested it but it should be possible to use the charm’s horizontal scaling feature with IBM Bluemix and Amazon S3.

With all of above in place we can deploy fully-configured, highly-available, apparmor-protected WordPress blogs in minutes. Our new charms allow for initial configuration to be performed automatically. Plugin subordinate charms support configuration options which will be added to the database; we even support plugin dependencies.

Full stack, starting from public facing side includes:

• squid-reverseproxy with apache2-subordinate for SSL termination
• wordpress and its subordinate services:
• apache2-subordinate
• wordpress-theme which pulls required theme from a bazaar branch. The charm supports different methods like git url or zipfile bundled with the charm
• wordpress-plugin for openstack-objectstorage. It installs and configures swift backend allowing easy horizontal scaling as soon as juju finishes deployment
• wordpress-plugin subordinate services for OpenID, Launchpad (and Launchpad Teams) integration plugins. They allow us to use Ubuntu SSO accounts to log in to our blogs and control permissions using Launchpad team membership
• mysql database

All of the above also have relations with nrpe-external-master and landscape services for monitoring and system management. Where needed subordinate charms take care of log archiving to Swift and backups.

High level diagram of the solution looks like this:

What does it mean in practice?

In Canonical we use mojo (https://mojo.canonical.com) to deploy all new services. Once we write a spec (set of files describing full deployment) we can run:

mojo run  ${MOJO_WORKSPACE}

which will deploy and configure all charms and add relations. Then it will run all nagios checks, ensure there are no hook errors and that all instances are up and finally perform end to end health checks ensuring our new blog is up and running. No human interaction needed.

This end to end, deployment and verification is very useful during development as well. After making charm changes, we rerun mojo to ensure our code has no unexpected side effect and that the end result is still what we want.

So are there any situations where one should not use the new charms? Yes. We optimized for security over convenience. Because of that WordPress has very limited access to the host operating system’s filesystem. Most notably it will not be possible to install plugins and themes using WordPress’ admin control panel. Instead plugins and themes must be added to the environment using juju for example:

juju deploy wordpress-plugin wordpress-plugin-wordpress-importer
juju set wordpress-plugin-wordpress-importer plugin_name=wordpress-importer
juju set wordpress-plugin-wordpress-importer code_uri=https://downloads.wordpress.org/plugin/wordpress-importer.0.6.1.zip
juju add-relation wordpress wordpress-plugin-wordpress-importer

We think this trade off is well worth it, but we understand there are situations where the charms will not be suitable.

Another thing to note is that horizontal scaling requires access to OpenStack object storage. Integration with Amazon S3 should work as well but we have not tested it. The WordPress charm will of course work in all clouds (and on bare metal through MAAS) but without object storage it will not scale horizontally.

Here are a few examples that demonstrate how common tasks can be performed using our new charm.

To have juju automatically configure our blog we can create wordpress.yaml file:
wordpress:

code_location: lp:~canonical-sysadmins/wordpress/4.0.1
blog_hostname: myblog.example.com
akismet_key: abc123
initial_settings: |
	user_name: admin
	admin_email: user@example.com
	weblog_title: My New Blog

And then deploy like this:

juju deploy wordpress --config=myconfig.yaml

If a new version of WordPress version comes out, we can upgrade the code and run schema updates by:

juju set wordpress code_location=lp:~canonical-sysadmins/wordpress/4.1

Git URLs work too:

juju set wordpress code_location=https://github.com/project/repo.git

Web designers have prepared a new theme revision. We can update it on all WordPress servers by:

juju set wordpress-theme theme_url=git://example.com/new-theme-branch

We want to disable and remove “WordPress Importer” plugin:

juju destroy-service wordpress-plugin-importer

We are getting more traffic so we want to add 3 extra wordpress servers:

juju add-unit -n 3 wordpress

The charms described are located here:

cs:~canonical-sysadmins/trusty/wordpress-services-0
cs:~canonical-sysadmins/trusty/wordpress-plugin-0
cs:~canonical-sysadmins/trusty/wordpress-theme-0
cs:~canonical-sysadmins/trusty/apache2-subordinate-1
cs:~canonical-sysadmins/trusty/squid-reverseproxy-0