#StopMullware — On the Security of 27% of the Websites on the Internet
This is the story of how I failed to proactively prevent every WordPress blog on the Internet from ever being conscripted into a Mirai-like botnet.
Who are you anyway? (Feel free to skip this part.)
I recently proposed a successful RFC to make libsodium a core extension in PHP 7.2, which will make secure and usable cryptography more accessible to web developers the whole world over. However, I’ve been working hard to make the Internet more secure for years, starting with the tools and frameworks that developers rely on in their day-to-day.
To be explicit: this Medium post is purely personal and does not necessarily reflect the opinions of my employer (otherwise it would obviously be posted on the company blog instead). If anyone takes issue with its contents, take it up with me personally.
Automatic Updates and Automattic Software
If you take nothing else away from this post, let it be:
- Automatic updates are a damn good idea. They reduce the feedback loop on security updates and prevent days-old vulnerabilities from being exploited in the wild.
- If automatic updates are implemented poorly, however, they can totally undermine the security gain.
In the case of Automattic’s most popular software project, WordPress (which has provided automatic updates since 3.7), there is no offline digital signature of the update files. If anyone can compromise their update server, they can infect every WordPress blog on the Internet (which is estimated to be 27% of all websites) with a malicious update for free.
The problem of secure source code delivery is a tricky one, and when you automate the whole process it becomes very attractive for attackers. At a minimum, the following measures should be taken:
- Transport Layer Security (HTTPS — HTTP over TLS a.k.a SSL, Noise, etc.)
- Offline Cryptographic Signatures
WordPress’s update server does support HTTPS. However, they do not have any sort of code-signing or verification mechanism in place.
HTTPS alone is not adequate. Just because you’re talking to the right server does not mean that the file you’re downloading hasn’t been tampered with.
That’s where offline code signing comes into play: The key that can produce a valid signature for a file isn’t stored on the server (only the file itself and a valid signature are), so even if the server gets hacked, attackers can’t just add trojan horse malware to the file.
More urgently, this attack already almost happened late last year.*
Naturally, being as interested in online security as I am, I had set out to solve that problem once and for all.
Interlude: Secure Cryptographic Signatures
There are a lot of different kind of digital signature algorithms to choose from, and I don’t want to turn this into a cryptography engineering lecture, but your only real options are:
- Ed25519 / Ed448 (RFC 8032)
- Deterministic ECDSA (RFC 6979)
If you propose a solution in 2017 that still uses RSA, cryptographers will look at you funny.
WordPress is written in PHP (and they provide support all the way back to 5.2) and the main interface for public-key cryptography in PHP is OpenSSL.
The OpenSSL extension to PHP only supports RSA, even when OpenSSL itself has support for elliptic curve cryptography. Either way, it was not deemed acceptable by the WP contributors I communicated with.
Even if newer versions of OpenSSL and/or PHP were to add support for elliptic curve cryptography, WordPress’s insistence on supporting abandoned versions of PHP (which no longer receive security updates) will in turn prevent them from ever using these new features.
These constraints left me with only two options:
- Levy a complaint about the version requirement that will fall on deaf ears.
- Write a pure-PHP implementation of modern cryptography that supports PHP 5.2 (because WordPress does) and use that to solve this problem.
And so, over the course of a few months, I wrote sodium_compat and submitted patches that used sodium_compat to add Ed25519 signature verification to WordPress’s automatic updater.
Early on, WordPress member Dion Hulse told me they couldn’t accept sodium_compat until it had been audited by a third party that specializes in software security and cryptography. I acknowledged this requirement in my first public statements about sodium_compat, before the first draft was even finished.
From the beginning, the plan had been:
- See if Automattic or WP.org (the primary beneficiaries of sodium_compat) would support the cost of an audit. Audits are pricey, so it’s completely understandable if they wouldn’t want to pitch in here.
- If Automattic didn’t help out, I would crowd-source it (through Jim Denaro who provided escrow services for the #isTouchIDHackedYet crowd-sourced bug bounty program), since something like sodium_compat is useful to the PHP community at large, not just WordPress.
- If sodium_compat passed its audit, it would be included inside a patch that would bolster the security of WordPress by ensuring updates were signed by the WordPress.org team.
So, to recap: my plan was to do all the work to make sure this infrastructure security problem was solved for them for free (regardless of whether or not they were willing to pay for the independent third-party review that they required in order for my solution to be accepted at all).
So you can imagine my shock yesterday when Dion Hulse informed me that Matt Mullenweg, CEO of Automattic and lead WordPress developer, ordered him to not do any work on package signing, because it wasn’t one of Matt’s goals for 2017 (which consists entirely of: Customizer, Editor, and REST API).
Maybe in 2018. But I’m not holding my breath.
You Have Every Reason to Be a Little Alarmed Right Now
A security enhancement that would have prevented any attacker capable of compromising api.wordpress.org from immediately winning 27% of the websites on the Internet with virtually no additional effort was blocked because Matt Mullenweg doesn’t consider security a priority.
Even if sodium_compat were audited tomorrow, Mullenweg refuses to allow the WordPress development team focus on package signing.
Let’s suppose an attacker was able to pull off such an attack tomorrow, and used it to build a Mirai-like botnet to use for DDoS attacks.
I doubt any network could withstand the bandwidth of every WordPress site being unloaded on them at once. (And chances are, they’ll also disable the automatic update feature to prevent the WordPress team from undoing their damage.)
The WordPress team has shown that they are not responsible enough to govern their impressive ownership of the Internet (with the exception of some folks powerless to correct the organization’s course). This act of negligence will put the rest of the web in harm’s way.
Thus, we are left with two options:
- Tell Automattic that security should be a top priority, since it clearly isn’t for them.
- Uninstall WordPress everywhere. Migrate as much of the Internet towards a platform whose custodians care about security. If nothing else, we can limit the impact for when WordPress does go nuclear.
I propose that we refer to any software that is marked by willful negligence towards security as mullware, in dishonor of Matt Mullenweg.
Tell your friends. #StopMullware today.
What’s going to happen to sodium_compat?
I don’t know yet.
The main purpose of said project was to make WordPress more secure, but that’s obviously not going to happen now.
There are still a lot of non-WordPress PHP projects that can benefit from modern elliptic curve cryptography. For their sake, I’m still strongly inclined to pursue an independent third party cryptography audit, and attempt to crowd-fund the cost. (The plan was to proceed with or without Automattic/WP’s financial support.)
However, knowing that WordPress’s lead developer simply doesn’t care about security and refuses to accept what was offered to him has robbed me of any enthusiasm for the short term.
Finally, I’d just like to say I’m sorry to the Internet as a whole.
I don’t know what I could have done differently to get the Automattic decision-makers on board with improving the security of their products, and consequently of a sizable chunk of the world-wide web.
I don’t know if anything can be done today or tomorrow that will cause Automattic to reverse their course. I personally think they’re stubborn enough to go down with their ship, no matter how preventable the wreck may be.
But I do know for sure that reducing the amount of mullware on the Internet will limit the potential damage for when WordPress’s update server does get compromised. If we push hard enough, we may still have an Internet at the end of the day too.
*More precisely, Wordfence discovered one such vulnerability last last year in api.wordpress.org that could have allowed an attacker to serve a malicious automatic update like I’ve been discussing.