A letter from Amazon

moxie0 on 01 May 2018

Last week, we received the following email from Amazon:

History

Direct access to Signal has been censored in Egypt, Oman, Qatar, and UAE for the past 1.5 years. These countries attempt to block Signal by blocking connections to Signal servers from all ISPs.

Like most modern services, Signal does not have a single static IP address that ISPs can filter. In cloud environments, IP addresses can change over time as load balancers scale up and down, and the addresses aren’t even always dedicated to a single endpoint. Amazon CloudFront, for example, may terminate requests on the same IP for any number of services that wish to distribute content on their CDN. This can make it more difficult for a censor to identify traffic based on IP address alone.

Unfortunately, a TLS handshake fully exposes the target hostname in plaintext, since the hostname is included in the SNI header in the clear. This remains the case even in TLS 1.3, and it gives a censor all they need.

However, several cloud environments were built with an idiosyncrasy that allowed us to work around this TLS metadata problem. Google and Amazon built their TLS termination layer separately from their request processing layer, such that it was possible to create what looked like a TLS connection for domain A with a request that would actually be received and processed by domain B. This is known as “domain fronting.”

When access to Signal was originally censored in Egypt, Oman, Qatar, and UAE, we responded by deploying domain fronting in those countries through Google App Engine. This meant that to block Signal, those countries would also have to block google.com. That was not a step those countries were willing to take, and as a result Signal has been usable there for the past 1.5 years, even though direct access remains blocked. This required no configuration from users; they could simply install the app and use it as normal.

Iran

Direct access to Signal has also been blocked in Iran for the past 3+ years, but it was not possible to use the same domain fronting technique there. In an apparently unique interpretation of US sanction law, Google does not allow any requests from Iran to be processed by Google App Engine. Requests would get past Iranian censors, but then Google themselves would block them.

In early 2018, a number of policy organizations increased pressure on Google to change their position on how they were interpreting US sanction law so that domain fronting would be possible from Iran. Sadly, these lobbying efforts seem to have had the opposite effect. When Google’s leadership became more aware of domain fronting, it generated internal conversations about whether they wanted to put themselves in the situation of providing cover for sites that entire countries wished to block.

A month later, we received 30-day advance notice from Google that they would be making internal changes to stop domain fronting from working entirely.

AWS

With Google no longer an option, we decided to look for popular domains in censored regions that were on CloudFront instead. Nothing is anywhere near as popular as Google, but there were a few sites that used CloudFront in the Alexa top 50 or 100. We’re an open source project, so the commit switching from GAE to CloudFront was public. Someone saw the commit and submitted it to HN. That post became popular, and apparently people inside Amazon saw it too.

That’s how we got to the above email. Although our interpretation is ultimately not the one that matters, we don’t believe that we are violating the terms they describe:

1. Our CloudFront distribution isn’t using the SSL certificate of any domain but our own.
2. We aren’t falsifying the origin of traffic when our clients connect to CloudFront.

However, in the time-honored tradition of sharing unpopular news late on a Friday afternoon, a few days ago Amazon also announced what they are calling Enhanced Domain Protections for Amazon CloudFront Requests. It is a set of changes designed to prevent domain fronting from working entirely, across all of CloudFront.

Future

With Google Cloud and AWS out of the picture, it seems that domain fronting as a censorship circumvention technique is now largely non-viable in the countries where Signal had enabled this feature. The idea behind domain fronting was that to block a single site, you’d have to block the rest of the internet as well. In the end, the rest of the internet didn’t like that plan.

We are considering ideas for a more robust system, but these ecosystem changes have happened very suddenly. Our team is only a few people, and developing new techniques will take time. Moreover, if recent changes by large cloud providers indicate a commitment to providing network-level visibility into the final destination of encrypted traffic flows, then the range of potential solutions becomes severely limited. If you’d like to help, we’re hiring.

In the meantime, the censors in these countries will have (at least temporarily) achieved their goals. Sadly, they didn’t have to do anything but wait.