NSA files decoded: Edward Snowden's surveillance revelations explained

By EWEN MACASKILL and GABRIEL DANCE
Produced by FEILDING CAGE and GREG CHEN
Published on November 1, 2013

When Edward Snowden met journalists in his cramped room in Hong Kong's Mira hotel in June, his mission was ambitious. Amid the clutter of laundry, meal trays and his four laptops, he wanted to start a debate about mass surveillance.

He succeeded beyond anything the journalists or Snowden himself ever imagined. His disclosures about the NSA resonated with Americans from day one. But they also exploded round the world.

For some, like Congresswoman Zoe Lofgren, it is a vitally important issue, one of the biggest of our time: nothing less than the defence of democracy in the digital age.

But the intelligence agencies dismiss such claims, arguing that their programs are constitutional, and subject to rigorous congressional and judicial oversight. Secrecy, they say, is essential to meet their overriding aim of protecting the public from terrorist attacks.

The debate has raged across time zones: from the US and Latin America to Europe and to Asia. Barack Obama cancelled a trip to Moscow in protest at Russian president Vladimir Putin's protection of Snowden. Brazilian president Dilma Rousseff cancelled a state visit to Washington in protest at the US spying on her. Bolivian president Evo Morales's plane was forced down in Vienna amid suspicion that Snowden was being smuggled out of Russia.

In Germany, a "livid" Angela Merkel accused the US of spying on her, igniting a furore that has seen the White House concede that new constraints on the NSA's activities may be necessary. Meanwhile, in Britain, prime minister David Cameron accused the Guardian of damaging national security by publishing the revelations, warning that if it did not "demonstrate some social responsibility it would be very difficult for government to stand back and not to act".

Much of the NSA’s defence is that the public should be unconcerned, summed up by the dictum: “If you have nothing to hide, you have nothing to fear.” But civil liberties groups such as the Electronic Frontier Foundation and the American Civil Liberties Union warn that surveillance goes well beyond what Congress intended and what the US constitution allows.

Cell phones, laptops, Facebook, Skype, chat-rooms: all allow the NSA to build what it calls ‘a pattern of life’, a detailed profile of a target and anyone associated with them.

And the number of people caught up in this dragnet can be huge.

Faced with growing public and political concern over the quantities of data it is collecting, the NSA has sought to reassure people, arguing that it collected only a tiny proportion of the world’s internet traffic, roughly equivalent to a “dime on a basketball court”. But in reality, that is still a huge amount of data. The Library of Congress, one of the biggest libraries in the world, gathers 5 terabytes a month. The NSA sucks up much, much more.

The NSA say it needs all this data to help prevent another terrorist attack like 9/11. In order to find the needle in the haystack, they argue, they need access to the whole haystack.

Snowden recognises the value of the NSA in counter-terrorism, but thinks the spy agency has dangerously over-reached itself. He is a fugitive from US law, in exile in Russia. But the debate he wanted to start when he decided to become a whistleblower is now happening.

One unseen consequence of the Snowden disclosures is the entry of the term ‘metadata’ into common usage. This is information about the time and location of a phone call or email, as opposed to the contents of those conversations or messages. The distinction forms the crux of the debate over the proper scope of NSA surveillance.

The first Snowden document to be published by the Guardian was a secret court order showing that the NSA was collecting the telephone records of millions of US customers of Verizon, one of America's largest telecoms providers.

It is this program that has dominated US political debate since then. Early in October, Senator Dianne Feinstein, the chair of the Senate intelligence committee, wrote in USA Today: "The call-records program is not surveillance. It does not collect the content of any communication, nor do the records include names or locations. The NSA only collects the type of information found on a telephone bill: phone numbers of calls placed and received, the time of the calls and duration."

But privacy activists critical of the NSA surveillance program vehemently disagree, arguing not only that the collection is based on a legal interpretation that goes way beyond what Congress allowed, but also that metadata includes personal information, which can build a more detailed profile even than listening into content.

Much of what the NSA does is of value to America and its friends round the world — even those it snoops on. The documents show the NSA providing vital information to American and allied forces in Afghanistan, defending the country against cyber attacks, snooping on Mexican drug cartels and helping break up worldwide criminal gangs involved in credit card theft.

Since the Snowden disclosures began, the NSA and the Obama administration have justified the agency’s programs by claiming they have been crucial to ‘successes’ in counter-terrorism.

The NSA, in its defence, frequently argues that if today’s surveillance programs existed before 9/11, it might have been able to stop those attacks. But this, too, is a matter of dispute. The intelligence agencies had a lot of capability before 9/11, and did pick up vital information, but failed to share it with one another or join up the dots.

Baker argues that the NSA has learned from its mistakes.

But exactly how successful the bulk collection of US data has been in preventing terrorist attacks since 9/11 is a matter of dispute.

In the immediate wake of the early NSA revelations, the agency’s director, General Keith Alexander, claimed the NSA surveillance had contributed to the prevention of 54 plots. But that number has been picked apart by the US media and Congress, forcing the NSA to revise it down. ProPublica have factchecked the 54 plots claim here and could only find evidence of four.

Eventually, deputy NSA director John Inglis conceded that, at most, one plot — which he has not specified — might have been disrupted by the bulk phone records program alone.

Two factors opened the way for the rapid expansion of surveillance over the past decade: the fear of terrorism created by the 9/11 attacks and the digital revolution that led to an explosion in cell phone and internet use.

But along with these technologies came an extension in the NSA’s reach few in the early 1990s could have imagined. Details that in the past might have remained private were suddenly there for the taking.

NSA is helped by the fact that much of the world’s communications traffic passes through the US or its close ally the UK – what the agencies refer to as “home-field advantage”. The NSA has its own cable-intercept programs tapping traffic flowing into and across the US. These operate mainly under four codenames — BLARNEY, FAIRVIEW, OAKSTAR and STORMBREW — and are collectively known as Upstream collection.

The Snowden documents show that the NSA runs these surveillance programs through “partnerships” with major US telecom and internet companies. Some of these relationships go back decades, others are more recent, in the wake of 9/11 and with the growth of the internet.

The division inside the NSA that deals with collection programs that focus on private companies is Special Source Operations, described by Snowden as the “crown jewels” of the NSA.

In one top document, published here for the first time, SSO spelled out the importance of these commercial relationships which come under the heading “Corporate Partner Access”.

In bald terms, it sets out its mission: “Leverage unique key corporate partnerships to gain access to high-capacity international fiber-optic cables, switches and/or routes throughout the world.”

As well as fiber-optic cables in the US, the NSA has access to data gathered by close intelligence partners such as Britain’s GCHQ.

The Snowden documents revealed the existence of Tempora, a program established in 2011 by GCHQ that gathers masses of phone and internet traffic by tapping into fiber-optic cables. GCHQ shares most of its information with the NSA.

As well as its upstream collection programs, the NSA also has Prism, which, according to the Snowden documents, is the biggest single contributor to its intelligence reports. It is a “downstream” program – which means the agency collects the data from Google, Facebook, Apple, Yahoo and other US internet giants. One slide claims the agency has “direct access” to their servers, but this has been hotly disputed by the companies, who say they only comply with lawful requests for user data.

When the Guardian and the Washington Post revealed the existence of Prism the companies denied all knowledge of it and insisted that any co-operation with the intelligence agencies was compelled by law.

The names of many of the NSA’s “corporate partners” are so sensitive that they are classified as “ECI” — Exceptionally Controlled Information — a higher classification level than the Snowden documents cover.

But some of the internet companies are named in the Special Source Operations briefing on Corporate Partner Access. A graphic comparing weekly reports involving the companies lists some of the Prism providers. Other companies on the list are protected by ECI covernames. Artifice, Lithium and Serenade are listed in other documents as covernames for SSO corporate partners, while Steelknight is described as an NSA partner facility.

This is the first time that data giving a sample of the number of intelligence records being generated per company has been published. It shows that over the period shown, June to July 2010, data from Yahoo generated by far the most NSA intelligence reports, followed by Microsoft, and then Google. All three companies are fighting through the courts to be allowed to release more detailed figures for the numbers of data requests they handle from US intelligence agencies.

Not all companies have complied. Ladar Levison, the founder of Lavabit — a small, secure email provider used by Snowden — suspended operations in August rather than comply with a warrant that would have allowed the US government access to the data of all Lavabit’s 400,000 customers.

In a statement defending its surveillance programs, the NSA said: “What NSA does is collect the communications of targets of foreign intelligence value, irrespective of the provider that carries them. US service provider communications make use of the same information super highways as a variety of other commercial service providers. NSA must understand and take that into account in order to eliminate information that is not related to foreign intelligence.

“NSA works with a number of partners and allies in meeting its foreign-intelligence mission goals, and in every case those operations comply with US law and with the applicable laws under which those partners and allies operate.”

But some members of Congress, such as Lofgren, who represents a Silicon Valley district, are unconvinced. She warns that the programs not only undermine individual privacy, but threaten the reputations of major American telecom and internet companies.

Millions of Americans struggling to get health insurance through Obamacare’s new health exchanges are entering some of their most intimate details into computer systems.

The technology they rely on to keep that information secure — along with their emails, online shopping, banking and more — is encryption. But your data may not be as secure as you might hope.

Encrypting a message involves scrambling it through a combination of a randomly-generated key and mathematical jumbling. The NSA and its UK counterpart GCHQ regard this as the biggest threat to their ability to view the vast quantities of communications data they collect.

Internet companies have given assurances to their users about the security of communications. But the Snowden documents reveal that US and British intelligence agencies have successfully broken or circumvented much of online encryption.

Much of this, the documents reveal, was not done through traditional code-cracking, but instead by making deals with the industry to introduce weaknesses or backdoors into commercial encryption – and even working to covertly undermine the international standards on which encryption relies.

Computer security experts say that by doing this in their quest to access ever more data, the intelligence agencies have compromised the computers of hundreds of millions of ordinary internet users, and undermined one their other key priorities – protecting the US and UK from cyberattacks.

So is all encryption broken? Snowden, in a question-and-answer session on the Guardian website in June, said that much of the encryption is weak, so the NSA can frequently find ways round it, but there are strong crypto systems that can still be relied on. Given that Snowden was inside the system until May, he should know.

Levison, the founder of secure email provider Lavabit, is facing a court case because he closed his company rather than hand over encryption keys.

The publication of the Verizon phone records order had one immediate political impact. It revealed that at a Senate committee hearing in March 2013, the director of national intelligence, James Clapper, had given misleading testimony. He was asked by Senator Ron Wyden whether the NSA collected “any type of data at all on millions or hundreds of millions of Americans”. Clapper’s reply: “No, sir”.

Forced to revise his answer after the Guardian published the document in June, Clapper at first said that he had given “the least untruthful answer” possible in a public hearing. But then it emerged that Wyden’s office had given the DNI 24 hours notice of the question, and an opportunity to correct the record shortly thereafter. Clapper changed his account to say that he had simply forgotten about collection of domestic phone records.

The erroneous testimony sparked calls for Clapper’s dismissal and has become a glaring example of failings in the oversight arrangements that are supposed to govern NSA surveillance programs.

The Snowden disclosures have led many on Capitol Hill and beyond to conclude that the political and legal mechanisms necessary to hold the NSA accountable in functioning democracy are no longer fit for purpose.

The Foreign Intelligence Surveillance Act of 1978 (Fisa) was intended to curtail the NSA’s ability to use its capabilities against Americans. It was passed as part of a backlash against one of the biggest controversies of that era: the unlawful surveillance by the intelligence agencies of US political activists, trade union leaders and civil rights leaders.

Fisa codified in law for the first time that the NSA was about foreign intelligence. If there was a suspicion about a spy or some agent of a foreign power operating in the US, the NSA and the FBI could apply for a warrant in a new surveillance court, the Fisa court.

But since then, according to Wyden, the way the laws work in practice by the intelligence agencies has become shrouded in secrecy.

The 2008 Fisa Amendments Act, renewed in 2012, allows for the collection of communications without a warrant, where at least one end of the communications is a non-US person.

The NSA legal basis — disputed — for bulk collection of Americans' phone data comes under a different law, section 215 of the 2001 Patriot Act. The Bush administration, in secret after 9/11, turned loose the NSA to collect bulk email records domestically.The NSA interpreted section 215 of the Patriot Act as allowing them to collect phone metadata in the US.

The Fisa court and its proceedings are secret, or at least they were until the Snowden revelations. Given this, it is nearly impossible to challenge its interpretation of the law. The government is the only petitioner before the court, with no advocates for privacy interests. The NSA argues that since that it is engaged in covert operations, it is hardly surprising that the court proceedings are secret.

In spite of Baker’s contention, the court has approved almost all government surveillance requests over the last 35 years.

The NSA is also subject to congressional oversight. But the limitations of this have become clearer over the past few months, with many members of Congress directly contradicting Obama’s persistent claim that they have signed off these programs, and insisting they had been totally unaware of the scope of the agency’s activities.

The politicians tasked with the greatest scrutiny are the Senate and House intelligence committees. Most of these — in particular Feinstein, the Senate intelligence committee chairwoman — have tended to be staunch defenders of the NSA.

The long-term sceptics, such as Wyden and his Senate colleague Udall, have been a lonely band. Even now, they believe they face an uphill struggle to achieve meaningful reform of the NSA.

The debate Snowden wanted is happening. That in itself is a major achievement.

But debate has expanded well beyond the confines of Capitol Hill, touching on individuals and groups throughout the US and elsewhere in the world.

One group feeling the immediate impact is journalists and their sources. The Snowden revelations have sent a chill through those reporters covering national security issues. If the NSA can easily gather details about who a reporter phoned or emailed, that sends a signal to whistleblowers that their anonymity can no longer be protected.

Public opinion is polarized over surveillance, but polls show a jump in concern over privacy in the wake of Snowden’s revelations. A Pew poll at the end of July found that for the first time in a decade, the majority of Americans are more concerned about the government infringing on their civil liberties than about a potential terrorist attack.

The shift is reflected in the change in attitudes over the past two years on a series of privacy issues.

In the end, it may be through the courts rather than Congress that genuine reform may come. Privacy groups such as the Electronic Privacy Information Center and the Electronic Frontier Foundation launched lawsuits that have led to disclosure of hundreds of pages of Fisa rulings on Section 215. GCHQ and NSA surveillance is facing a legal challenge at the European court of human rights from Big Brother Watch, English PEN and Open Rights Group.

Silicon Valley is also taking action through the courts. Google, Microsoft and Yahoo, facing a backlash from their users in the US and overseas over mass surveillance, are fighting to be allowed to be more transparent about their dealings with the intelligence agencies. These companies, along with Facebook, Apple and AOL have also written to the Senate intelligence committee demanding reform.

The political fallout from the NSA revelations began slowly, but in July it became dramatically apparent in Congress. The occasion was a vote in the House on Republican Justin Amash’s amendment to curtail funding for the NSA’s bulk collection of phone records for millions of Americans.

The amendment only narrowly failed to get through, with 205 in favour and 217 against. Support for change brought conservatives and liberals together in an unusual alliance.

There are now several major pieces of legislation going through Congress that would introduce at least some reform of the NSA. Among those, the one backed by Feinstein and passed by her committee is the least radical, offering proposals for greater transparency but basically maintaining the status quo. The bulk collection of Americans’ phone call data would be enshrined in US law.

More far-reaching is the proposed Intelligence Oversight and Reform Act, with bipartisan support from senators Wyden, Udall, Richard Blumenthal and Rand Paul. It would ban the collection of internet communication data; close loopholes that allow snooping on Americans without a warrant; reform the Fisa court; and provide some protection for companies faced with handing over data to the NSA.

Another bipartisan bill, backed by high-ranking senator Patrick Leahy and congressman Jim Sensenbrenner, who was one of the architects of the Patriot Act, would also end bulk collection of phone records. As part of reform of the Fisa court, it is proposed that a special advocate be created.

Video: Bob Sacha
Production: Kenan Davis, Nadja Popovich, Kenton Powell, Ewen MacAskill, Ruth Spencer, Lisa van Gelder
Additional Production: Spencer Ackerman, Kayla Epstein, Paul Lewis, Amanda Michel, Katie Rogers, Dominic Rushe