While Parliament continues to consider a dramatic expansion of national security powers in Australia, the UK and the US are on divergent approaches to the core issues around mass internet surveillance.

The joint committee on intelligence and security (JCIS) has yet to release its report on over 40 proposed extensions of national security laws put forward by then attorney-general Nicola Roxon last year. Despite the wide range of the proposals, the committee has had to primarily focus on one of the most controversial: forcing ISP and other service providers to retain data on individuals’ internet usage.

The same issue has arisen, albeit in different contexts, in the US and the UK. Along with Canada and NZ, Australian intelligence agencies collaborate closely with US and UK agencies in an Anglophone grouping known as the Five Eyes.

First to the UK, where the Cameron government’s plans to establish data retention (quickly branded a “snooper’s charter”) were derailed in late April when the Deputy PM, Liberal Democrat Nick Clegg, said his party wouldn’t support it. The plan had been pushed by Conservative Home Secretary Theresa May.

May’s Communications Data Bill went well beyond the European Union data retention directive, which also applies in the UK. The EU directive itself is now under serious attack, with some EU bodies claiming it infringes basic rights to privacy and a number of governments, most significantly Germany, refusing to legislate it or address court decisions overturning it because it is unconstitutional. The EU directive was the model for Roxon’s data retention proposal here and is limited to “telecommunications data” about internet usage, and not “content data”about what sites have been visited.

But under the EU directive, telecommunications companies have mistakenly retained full internet activity histories and telecommunications data has been used to hunt down whistleblowers.

But May’s CDB would have extended beyond telecommunications companies to anyone carrying data, such as social media sites, and compelled them to generate usage data (as opposed to retaining data generated for billing purposes) or, if a site was offshore and beyond the reach of British law, compel intermediaries to record usage data.

In April, Clegg appeared to kill the CDB off, saying the bill failed to get the balance right between liberty and security (a rare example of a politician using the concept of “balance” to oppose more security). He also correctly noted that the UK establishing such a draconian system would set a “worrying international precedent”. Clegg’s opposition was welcomed not merely by his own party but by many Tories opposed to the bill.

But a part of the bill appeared to return to life last week in the Queen’s Speech, in which she said, “in relation to the problem of matching internet protocol addresses, my government will bring forward proposals to enable the protection of the public and the investigation of crime in cyberspace”.

This suggests a reduction in ambition by the Home Office (the equivalent of the Attorney-General’s Department), which like AGD has aggressively pushed for internet surveillance. Matching IP addresses with individuals is a core priority for AGD as well. In a confrontation with Greens senator Scott Ludlam at Estimates last year, AGD and Federal Opolice officers said the focus was on the “internet identifier [information that uniquely identifies a person on the internet] assigned to the user to the provider”. Agencies were less interested in knowing what addresses each person had visited, AGD Secretary Roger Wilkins and the AFP’s Neil Gaughan insisted, and more in finding out who was behind an IP address they had connected to a site such as, for example, a child abuse or terrorism website.

That approach differs from other regulators here, like the Australian Securities and Investments Commission, that want full, permanent records made of everyone’s internet browsing, but that has already been publicly dismissed by JCIS as an ambit claim.

However, there are significant problems with actually matching IP addresses to individuals — and AGD knows about them. Industry participants explained, during the course of AGD’s laborious consultations on data retention that they kept secret until forced to reveal them by JCIS, that the increasingly mobile nature of internet usage makes linking IP addresses and individuals problematic. And that’s before you get to the problem of encryption. Wilkins tried to wave away the problem of encryption during a JCIS hearing last year when he said “we’ll demand the encryption keys” for services like Tor — plainly not aware that Tor has no permanent encryption keys and they’re not known by Tor administrators anyway.

The UK security establishment was deeply unhappy with the defeat of the snooper’s charter, claiming they need to address the problem of how communications technology is rendering their surveillance systems obsolete. That’s the same challenge the FBI claims it is facing. We’ll look at how the Obama administration says it will respond to the problem of “going dark” tomorrow.