The Death of Secret Intelligence? Think Again

While the Ukraine war has seen an explosion in the collection and distribution of open source intelligence, the work of secret intelligence agencies remains as important as ever.

You’d be forgiven for thinking that the world of secret intelligence is no more. Ever since Russia’s invasion of Ukraine in February 2022, there has been a stream of opinion pieces arguing that traditional state-based agencies are struggling to keep up in the face of open source intelligence (OSINT). The volume and speed at which we can access publicly available information is a revolution that large secret intelligence agencies are struggling to cope with, or so we’re told. Whether it’s commercially sourced satellite imagery or social media posts, the US and UK intelligence communities are operating in an increasingly complex environment – that much is true.

But while it may seem trendy to claim the world of OSINT challenges state-based capability, the reality is that intelligence agencies are doing just fine. They really don’t need me to say this, but the UK and US intelligence communities still have much to bring to the table. This might seem to be a false dichotomy – we need both secret intelligence and OSINT to improve decision-making, but the secret world is often neglected in the drive to flag up new OSINT techniques. You’d be forgiven for thinking that in what might be called the first digital war, everything is open. It’s not. My message is a simple one: secret intelligence is still significant, and will be for some time.

Don’t get me wrong: when OSINT is good, it’s good. Think of the Bellingcats of this world, utilising largely open (and some ‘grey’) information to conduct a thorough assessment. The downing of MH17, the Skripal poisoning and investigations into Syrian use of chemical weapons all stand out. And no, they don’t need government to help. But when OSINT is bad, it’s very bad. More available information does not necessarily lead to the right answers. Too often, OSINT practitioners look at the sexy topics, when what we need is the mundane. The growth in wannabe OSINT analysts since 24 February 2022 has not necessarily added anything to wider knowledge. It just further complicates an already complicated information space, and creates a perfect opportunity for misinformation. Just look at recent events in Russia. Yevgeny Prigozhin’s mutiny led to a stream of commentary suggesting that this was 1905, 1917 or 1991 all over again. None of this was true, and analysts had to quickly change their conclusions as events evolved. Similarly, there has been a sudden growth in counteroffensive ‘experts’ who will regularly update maps and provide social media updates. Much of this is bad assessment, yet it is shared widely. And it can only get worse in the era of deepfakes.

OSINT is also not new, and has been used for a long time. Only the volume and ease of access has changed. In the 1990s, the UK Foreign and Commonwealth Office’s Reginald Hibbert argued that, of all the information collected on another country, over half came from ‘overt sources – newspapers, radio and TV broadcasts, journals, books, pamphlets, lectures’. Information obtained from privileged or ‘grey’ sources by diplomats, defence attachés and others formed another key part, leaving just 10–20% of information coming from truly secret intelligence. While the true figure is hard to guess at, estimates in the US suggest it is closer to 10%. For priority (often harder) targets where OSINT is less copious, percentages of secret information will be higher.

Too often, OSINT practitioners look at the sexy topics, when what we need is the mundane

Much earlier, writing in 1967, Cabinet Secretary Sir Burke Trend said open source and secret material was ‘indivisible’. Whitehall departments, he wrote, ‘are organised to exploit the overt sources of intelligence of interest to them. Information which cannot be obtained from overt sources must be sought from covert sources’. And the opening to Lord Butler’s 2004 review admitted that while decisions were made based on ‘many types of information’, secret sources were necessary. OSINT and secret intelligence go hand in hand. Without the open source, we don’t know what gaps need to be filled by the secret.

So, do we actually need secret intelligence? The answer is unequivocally, yes. ‘OSINT is not a substitute for satellites, spies or existing organic military and civilian intelligence capabilities’, says NATO’s 2001 Open Source Intelligence Handbook. This statement holds true.

There is a lot we don’t know about Western intelligence collection on Russia. But what we do know is that Western agencies are having successes. A big conclusion from the leak of US documents on Discord in April 2023 was that US intelligence had extensive knowledge of Russia’s planning. Intelligence sources told US media they were aware of Prigozhin’s rebellion from mid-June onwards, though the timing remained a mystery. Some European allies were taken by surprise.

Since the start of the war, US intelligence has been sharing ‘real time’ intelligence to help ‘inform and develop’ Ukraine’s response. Such intelligence has been ‘vital’ and ‘high end’, yet the full details will never be known. Recently, CIA Director Bill Burns has reaffirmed US commitments to share classified information with Kyiv. The UK has also provided its own intelligence support.

For the UK, as elsewhere, OSINT has provided a useful cover to push out intelligence for impact. Previously, intelligence sources had to be safeguarded at all costs. Today, Defence Intelligence (DI) openly refers to publicly available information. The daily intelligence ‘event’ of DI’s slides can be tedious at times, yet it provides a useful service in confirming aspects of a conflict that can be confusing. The open source world has, as General Sir Jim Hockenhull’s speech to RUSI in November 2022 made clear, forced a rethink. Yet, even then, OSINT only contributes ‘somewhere in the region of 20%’.

Though the volume and nature of OSINT certainly poses questions for modern-day intelligence agencies, it would be wrong to see its presence as throwing into doubt the latter's importance

The secret world matters for a number of reasons. Recently, I’ve been reminded of the work of Sir Michael Quinlan, former Permanent Under-Secretary at the Ministry of Defence, brought in to ask whether we needed secret intelligence at the end of the Cold War. Quinlan’s review, completed in 1994, is as relevant now as it was then. For Quinlan, secret intelligence ‘seeks – whether by unearthing specific nuggets of information, or by building up patterns of understanding to help our ability to interpret what we observe – to offset gaps, uncertainties and distortions in what we can find openly’. Quinlan also made the point that secret intelligence plays a role where information is ‘copious’, helping ‘sift the flow – to validate and calibrate’ sources. More importantly, there are always going to be hard targets – even in the digital age. Quinlan found that at a time when questions about intelligence and the Cold War ‘peace dividend’ were raised, there was still a significant need for secret intelligence.

Today, fruitful sources for OSINT can be closed off by rivals. Secret sources can be harder to detect and stop. Only state-based agencies can collect in the volume needed at times, and are mandated by law to carry out their tasks.

Assessment is another key function often lacking in the wider OSINT community. As Sir Simon Gass, the recently retired chair of the Joint Intelligence Committee, has said, ‘The sheer quantity of high-class open source presents a problem for policymakers in its own right … even if all you’re doing is distilling it down into a story for them, that in itself is important’. Whereas OSINT just exists, those involved in the collection, assessment and management of secret intelligence can identify gaps, re-task and tailor the intelligence process to fit national security proprieties.

Secret intelligence is often indivisible, as Trend said, from wider information flows into government. Though the volume and nature of OSINT certainly poses questions for modern-day intelligence agencies, it would be wrong to see its presence as a challenge that throws into doubt the latter’s importance. OSINT has always been there, and always will be. Secret intelligence remains as important as ever – it’s just that its impact is less noticeable than OSINT. The impact on the war in Ukraine is something that future generations will have to assess.