Even for a debate marked by lies, wilful misinterpretation, denialism and obfuscation, it was one of the dumbest contributions of all, a statement so utterly absurd that it stood out even in the discussion of data retention.

Jacinta Collins, Victorian Labor senator, rose at the start of the Senate debate on the government’s mass surveillance bill yesterday. Collins, a former shop assistants’ union hack, is not the brightest politician in Parliament, but she explained to the chamber how the government had done an appalling job of explaining the need for data retention to Australians. “The Australian people must be satisfied that in seeking to defend ourselves from crime and terrorism we do not trample upon the very rights and freedoms that characterise Australia as a free and open democracy,” Collins opined. “The Abbott government has failed this test.”

Damning stuff. Was this a prelude to Collins saying she couldn’t support something on which the government had so evidently failed? Er, no — her very next sentence was: “This failure has left the Australian people vulnerable to a scare campaign about data retention … the Greens Party have exploited this vulnerability ruthlessly. They have deliberately and irresponsibly misrepresented the facts for their own cynical political purposes.”

There you have it: the problem with data retention isn’t the failure to balance the need to protect Australians versus their basic rights and freedoms, it’s the way it allows the Greens to run a scare campaign.

In truth, though, it didn’t matter what Collins said — Labor is backing this ridiculous mass surveillance scheme that no one has been able to justify, even the most absurd elements, like jailing anyone who reveals that the police are targeting a whistleblower by going after a journalist’s metadata.

Let’s take a step back and think about why, given the quality of the debate about data retention has been so poor, and its advocates so incompetent, it is still on the verge of being passed into law. This is an issue that was rejected by the former government after a bungled effort to ram it through cabinet by the Attorney-General’s Department, on which current Attorney-General George Brandis publicly humiliated himself trying to explain what the scheme involved. Police forces have been unable to produce evidence backing the need for such a scheme, and the Attorney-General’s Department has treated the committee that examined the proposal with contempt on multiple occasions. And yet it will be legislated with bipartisan support at a time when the government is having immense difficulty getting key elements of its legislative program through.

Moreover, it is only the latest “tranche”, as George Brandis likes to call it, of a series of massive increases in the powers of intelligence and law enforcement agencies passed in the last nine months. This achievement points to the extraordinary power of the unelected and unaccountable national security elite to get their way in terms of both powers and resources inside government, and the success of the tactics they use to get their way.

1. Persistence. Politicians, parliaments and governments come and go. The security elite are always there. Just keep bowling up the same proposals with the same claims as new attorneys-general arrive and depart and Labor and the Coalition alternate in power. AGD has been pushing data retention, for example, since at least early 2008 to the then-new Labor government, eventually without success. It bided its time with the new Coalition government, but eventually found a compliant cipher in Brandis.

2. Use maximum secrecy. The national security elite, while bureaucrats, are not subject to the normal accountability that other bureaucrats are. They routinely invoke operational security as a reason to avoid the most basic parliamentary scrutiny via the few open forms of parliamentary accountability to which they are subject. The primary purpose of the Joint Committee on Intelligence and Security is actually to provide the kind of basic financial accountability that public service agencies are normally exposed to via Senate estimates, behind closed doors.

On data retention, AGD has relied heavily on secrecy in developing proposals, trying to ensure its industry consultation over the last seven years has been kept confidential and using any excuse possible to reject freedom of information requests and censor documents it has been forced to release. Even the most recent round of industry consultation on the metadata definition and cost of the scheme was conducted by AGD in secret, with the department refusing to share the information with JCIS even in camera, let alone publicly.

3. Refuse to engage. A persistent theme of how the national security elite operate is that they simply refuse to engage on issues where there is no evidence to back them up — a tactic that is rarely available to other bureaucrats. This was demonstrated in quite literal terms when Greens Senator Scott Ludlam asked AGD bureaucrats for evidence to back up their claims about data retention and warrant requirements and they simply refused to answer. Another instance involved the purported watchdog of the security elite, Inspector-General of Intelligence and Security Vivienne Thom, who refused in estimates to answer the most basic questions about the biggest intelligence scandal of the last decade, our spying on East Timor and the silencing of the whistleblower who revealed it.

4. Buy off the most powerful critics. The government’s last-minute efforts to shut down media criticism of the impact on journalism of data retention were an interesting example of a rare challenge to the national security elite. The mainstream media were virtually silent on data retention, for several years, until key figures belatedly realised it might have a significant impact on journalists themselves — at which point they launched high-profile attacks on it. As one of the most powerful lobby groups, and one capable of prompting genuine alarm among average Australians, the media could not be ignored, and the elite was forced to agree to some mostly fig-leaf protections for journalists, sufficient to pander to the media’s self-obsession without actually ameliorating the threat mass surveillance poses. Other industries that have an equal case for such treatment, but that are less powerful lobbyists — like the legal industry — have been unable to secure similar treatment. Ordinary citizens, of course, can be disregarded entirely.

5. Exploit terrorism and child abuse. Despite the low level of threat posed by both terrorism and child pornography to Australians, both feature prominently in the efforts of the national security elite to justify surveillance, with politicians briefed to insist only mass surveillance can enable terrorists or paedophiles to be stopped — despite regular demonstrations from here and overseas of terror attacks carried out or planned by people already known to security forces, and evidence from police themselves that data retention is little use in finding paedophiles. The real importance of terrorism and child abuse is the way voters react emotionally, rather than rationally, to both.

6. Convert enemies to friends. One of the untold stories of the data retention debate is how George Brandis and Malcolm Turnbull, once heavyweight opponents of data retention, have been turned into the two primary agents of its establishment. That is partly the consequence of their shift into government — both have thus gone from opponents of Labor’s data retention scheme to advocates of the Coalition’s data retention scheme. But for Brandis, at least, we know from his own words that the national security elite made a determined effort to scare him into compliance by bombarding him with “intelligence”: “As the minister within the Australian system with responsibility for homeland security, the more intelligence I read, the more conservative I become,” Brandis has said, reflecting the extent to which he had allowed himself to be converted from his previous proud civil libertarian position to one of enthusiastic, if incompetent, enforcer of a security state.

7. Exploit politicians’ technological ignorance. The JCIS hearings into data retention were marked by some extraordinary displays of ignorance by its members, and as Brandis demonstrated so vividly, few ministers have even a basic grasp of the detail of communications technologies and surveillance. This makes them all the easier both to scare and to assure as to the maximum effectiveness, and minimum intrusiveness, of surveillance technologies.

With such tactics, the powerful clique that exercises power out of sight within the government can finally toast success in its long campaign to impose mass surveillance.