Special Duty: A History of the Japanese Intelligence Community

  • Fall 2019
Special Duty: A History of the Japanese Intelligence Community

Material reprinted from Special Duty: A History of the Japanese Intelligence Community by Richard J Samuels. Copyright © 2019 by Cornell University. This excerpt from the preface is used by permission of the publisher.

FALL 2019 : précis Faculty Feature
Richard Samuels and his new book Special Duty
November 27, 2019

One often hears Japanese refer to their country as a unique small island trading nation, precariously dependent on imported raw materials and adrift in a hostile world. Apart from the fact that all nations claim to be unique, that Japan is not small economically or demographically, and that its dependence on imports is no greater than that of many other countries, there is some truth in this mantra. Japan’s neighborhood, and the world in which its businesses and citizens operate, have always been filled with threats. This has never been truer than it is today, when shifts in Tokyo’s relations with its colossal Chinese and nuclear-armed North Korean neighbors portend modification of relations with its powerful US ally. Japan’s intelligence officers have to judge the speed, trajectory, and certainty of transformations in the balance of power, and policy makers need to decide what measures to take to protect those businesses and citizens. In the decades of study of Japan’s evolving security community, virtually no sustained attention has been paid to its once expansive—and then atrophied—intelligence community. This community is atrophied no longer; a close look at its past, present, and future is overdue.

Japan was in ruins and its intelligence community was at its feeblest at the end of the Asia-Pacific War, a time of momentous institutional enhancement of the US intelligence community. In January 1946, when the triumphant President Harry S Truman created a National Intelligence Authority and the post of director of central intelligence to coordinate government-wide intelligence activities, he seized the chance to have some fun. The president gave his senior deputies black cloaks, mustaches, and wooden daggers. The fortified US intelligence apparatus—and those of its allies—were not always playgrounds for practitioners, but thanks more to novelists, screenwriters, cartoonists, and comedians than to scholars, “licensed skullduggery”—and the secret agents who practice it—became Cold War stereotypes and satirical fodder. Who did not appreciate James Bond, George Smiley, or Jason Bourne? And who was not amused by Boris Badenov, Natasha Fatale, Maxwell Smart, or Austin Powers?1

Of course, real spies have always been among us, many associated with the wisdom of our greatest leaders. According to Numbers 13, Moses sent spies into Canaan under God’s direct order to report on the land conditions. In Kings 2, the Assyrians drew on an extensive intelligence network during their invasion of the kingdom of Judah in 701 BC. Julius Caesar noted that the Gauls regularly interrogated travelers and merchants for information about distant lands to gain strategic advantage, which they thereupon squandered for lack of analytic skills.2 George Washington relied on spies during the Revolutionary War, and upon becoming president in 1790, he persuaded Congress to establish a “Contingent Fund of Foreign Intercourse,” a secret intelligence kitty that grew to 12 percent of the entire federal budget within three years.3 In 1861, President Abraham Lincoln personally recruited a southern businessman to provide intelligence to Washington.4 And as recently as the eve of America’s entry into World War II, President Franklin D Roosevelt was dispatching personal friends to gather information on war-torn Europe. 

By then—actually by the end of World War I—intelligence had already become as much a matter for professional bureaucrats as for spies and their derring-do. Indeed, our embrace of the exploits of secret agents belies both how difficult the intelligence business is and the deadly serious role it plays in national security affairs. Members of an intelligence community—shorthand for the network of collectors of adversaries’ secrets and analysts of threat—are in the business of helping decision makers manage uncertainty. They must separate potential and distant challenges from real and near ones in an environment in which their enemies’ intentions are often the most closely guarded of all secrets. They must separate what matters from what only seems to matter, to distinguish what is known from what is unknown, and to know what they do not know.5 Then, as if this were not difficult enough,they have to transmit their evaluations to decision makers who have multiple reasons to discount or misuse them. As one unnamed senior State Department intelligence official described his unit’s role, “A good day is when we prevent a bad policy decision from being made.”6 Walter Laqueur offers a fitting metaphor: intelligence is “the Cinderella of contemporary politics: long hours, unpleasant work, humiliation, lack of recognition, and no Prince Charming in sight.”7 Once we acknowledge that all this takes place in the context of existential threat, intelligence ceases to appear all fun and games, and getting its organization right becomes imperative. 

Our perceptions of the intelligence community have been shaped predominantly by American, British, Soviet, and Israeli espionage—by Ludlum’s CIA, le Carré’s MI6, The Americans’ KGB, or Reicher Atir’s Mossad—and we may have been led to believe that spies are less suave and resourceful elsewhere. This may be why few Japanese spies have been popularly associated with either wisdom or heroism in Western accounts, though some were both wise and heroic. Nor have many been the benign objects of satire. More often, their malign caricature abroad was formed out of a supposed orientalist capacity for treachery, such as this from the British historian Peter Elphick, who insisted that “the Japanese national psyche” explains why expatriate Japanese who were “required to serve as part of a subversive network [were] deeply honored they were serving their emperor” as spies. Then there is the purported native incapacity of Japanese to act independently, as in this from the US Strategic Services Unit immediately after the Asia-Pacific War: “Jap mentality is completely unsuited to listening post work. They are slow, cautious thinkers, and can never make a quick decision or take prompt action before thinking up a suitable reason or excuse [that is] sufficiently watertight to [protect] against loss of face.”8 Even the Chinese have waved the essentialist culture card at the Japanese. Chiang Kai-shek, the Chinese Nationalist leader who once served in the Imperial Japanese Army, reportedly declared that “everyone Japanese, both male and female, is a born spy.”9 The popular allure of ninjas notwithstanding, clearly there is some confusion abroad regarding whether the Japanese are or are not inherently adept at espionage. As we shall see, there was no inherent intelligence deficit preventing the Japanese intelligence community from expanding or from having its share of success during the first half of the twentieth century. 

Meanwhile, the joke at home in Japan has been about the bureaucracies, not the secret agents. In the standard Japanese narrative, during the Cold War Japan had no CIA but did have a “KGB”: Keisatsu (National Police Agency), Gaimushō (Foreign Ministry), and Bōeichō (Defense Agency). This speaks to a fundamental truth that will inform much of the analysis in this book: these separate government agencies—like those elsewhere within Japan and in intelligence communities abroad—seem to have forever been engaged in intense (sometimes petty) jurisdictional competition, captive in silos inhibiting coordination. The Japanese intelligence community, like the US and British ones, took a sharp bureaucratic turn—perhaps even earlier and more sharply than in Washington or London. In the Japanese narrative, more of the country’s heroes—many of whose photos are in this volume—were celebrated as military and government officials than as gallant national champions. They were patriots, of course, but in the first instance they were cashiered soldiers in the former Imperial Japanese Army who were joined by diplomats in the Ministry of Foreign Affairs (MOFA), crime fighters in the National Police Agency (NPA), economists in the Ministry of International Trade and Industry (MITI) and the Ministry of Finance (MOF), and lawyers in the Justice Ministry’s Public Security Intelligence Agency (PSIA). They belonged as much to their competing units as to the Japanese state. And many, of course, were politicians vying for power. Japan’s intelligence units were small, non-comprehensive, uncoordinated, underfunded, and, as a result of lingering political sensitivities (especially regarding the use of spies), unnecessarily baroque. They all operated in an environment of mutual distrust with limited central authority and even more limited public support. Kotani Ken, a leading historian of Japanese intelligence, tells us that the government “never succeeded in managing the central intelligence system effectively,” and even today, many observers simply throw up their hands and declare, “Japan has no intelligence community.”10 

So this term will be used with caution in these pages. And indeed, we can recognize that not all the problems encountered (or created) by the Japanese intelligence bureaucracies should be connected to the domestic structure of strategic policy making. Subordination to Washington also muted interest in developing Japan’s postwar intelligence community. During the Occupation—and even well after Japan regained sovereignty—its intelligencefunction was derivative, underdeveloped, and narrowly aimed at domestic enemies and foreign firms. The larger strategic horizon was monitored by its ally the United States. Resentment of Japan’s subservience to its US partner—what one intelligence journalist has called a persistent “master-servantrelationship” (shujū kankei)—never independently forced the shape and pace of Japanese intelligence reform, but it did become a more persistent problem than is normally acknowledged.11 Most Cold War Japanese intelligence and security professionals accepted that they had little choice but to accommodate to US power. As a result—and notwithstanding that there remain gaps in sharing and trust—there has been increasing integration of the two intelligence communities.

While accommodation to the preferences and practices of the US intelligence community was a defining feature of the first decades after the war, the Japanese intelligence community—like the military overall—was also stifled by clear and insistent public opposition to any practice redolent of wartime governance. Above all this meant that engaging in (or even debating the merits of ) intelligence—especially counterintelligence, but also counterterrorism—was problematic. Every plan, each discussion of the topic, raised hackles among those who feared (not without cause) that the Japanese could slide down a slippery slope back to unrestrained practices like domestic surveillance and foreign aggression that destroyed millions of lives and their nation.

After the Cold War, thoughtful Japanese national security strategists—in both the bureaucracy and the political class—took up intelligence reform with new energy. They began to tinker, reconceive, and, finally, to restructure Japan’s national security apparatus—and with it Japan’s intelligence community. These, then, are the transitions that are identified and followed in this volume: the expanding, accommodating, tinkering, reimagining, and reengineering of the intelligence community of one of the world’s great powers. It will be a story that shifts from a focus on individuals and their exploits to organizations and their competitions. We will discover how Japan was propelled on this century-long course, why reform was so constant and difficult, and what consequences this had for Japan’s national security. 

Like most policy change, in Japan or anywhere else, intelligence reform cannot be reduced to a single cause—at least not without sacrificing accuracy. Chapter 1 identifies three generic drivers that affect the shape, pace, and direction of intelligence reform. None is surprising. The first consists of shifts in the strategic environment. After all, threats and balances of power change. Consider, for example, how and why the OSS became the CIA as the Cold War set in after World War II and the United States found itself having to adjust to being one of two superpowers in a suddenly bipolar world. For their part, Japanese strategic thinkers have always been sensitive to Japan’s geostrategic circumstances, and often have responded with intelligence reform at moments of strategic uncertainty. The Foreign Ministry built intelligence capabilities to undermine tsarist Russia at the turn of the twentieth century, the imperial military began to rely heavily on private intelligence sources in China in the 1920s, the police strengthened counterintelligence and counterterrorism capabilities in the 1920s at the moment when mass parties were mobilizing (in some cases with foreign support), and every agency beefed up its collection and analysis capabilities in the 1930s when war with the United States seemed imminent. 

After losing that war, the Japanese intelligence community endured an extended period of subordination to the United States from which it only recently has begun the delicate process of freeing itself by enhancing indigenous capabilities without denying itself the benefits of US intelligence support. As I have already briefly noted—and as I examine in detail in chapters 3 and 4 of this book—Washington’s domination of Japan’s intelligence community throughout the Cold War generated considerable resentment in the Japanese security community. The US Department of Defense reportedly blocked Japanese acquisition of surveillance satellites for a long time, and Japanese Ground Self-Defense Force intelligence units were run under code names that were kept secret from their US partners. Remnants of this resentment persisted after the Cold War, affecting some more than others, but the depth of this resentment varied and never rivaled geostrategy as an independent driver of intelligence reform. US frustration with Japanese intelligence leaks generated pressure from Washington that was as problematic for the Japanese intelligence community as the leaks themselves. 

Strategists are fully aware that the global balance of power has shifted several times since the end of the Cold War, and many are quite naturally concerned that continued dependence for security and intelligence on a United States in relative decline renders Japan vulnerable in new ways. As we shall see, this was evident to many well before Donald Trump’s campaign and election in 2016 elevated their concerns. Nearly a decade earlier—even before China’s military threat became palpable and North Korea’s acquisition of nuclear weapons openly challenged the US position—a former chief of MOFA’s Intelligence and Analysis Service argued that Japan could not effectively reform its intelligence community until it realized that Washington would not necessarily provide for Japan’s security and came to grips with the need to formulate a truly independent diplomatic strategy.12 Just as Prime Minister Abe Shinzō was preparing to reengineer Japan’s foreign policy and intelligence system in 2013 by creating a National Security Council (NSC), three Diet representatives from different parties—one of whom would become Abe’s foreign minister in 2017—issued a vigorous call for an independent Japanese intelligence service, justifying it with this assessment, rhetorical question, and prescription: 

Japan’s diplomacy and national security have never been in such a tight fix. America’s relative power is declining and China’s military rise, as well as its expanding claims in the ocean, are striking. . . . Is Japan responding effectively to the historic shift in world order? . . . Even if an NSC is established, there is still a missing piece—a foreign intelligence [unit].13

The second driver of intelligence reform has been technological change. The most prominent intelligence-related technologies have had to do with the way intelligence is collected: human intelligence (HUMINT), radio and other signals intelligence (SIGINT), and image intelligence (IMINT) are the most widely known and are all widely practiced. Cyber-based intelligence harvesting is just the latest tool to which intelligence communities have to adjust. The Japanese intelligence community has been an active, indeed voracious, technology follower for well more than a century. The imperial military first experimented with aerial reconnaissance balloons in 1877, during the Satsuma Rebellion, and their first operational use came in 1904, during the Russo-Japanese War. It was during that conflict when the Imperial Japanese Navy stood up its first SIGINT unit and broke Russian codes. By that time, and for decades after, Japanese agents—some official, many not—were engaged in extensive espionage activities across Eurasia using advanced tools, including encrypted communication devices. Even so, Japanese messaging proved vulnerable to interception. Washington was privy to Japan’s negotiating positions during the Washington Naval Conference in 1921–22 and used its SIGINT advantages to intercept much of Japan’s military and diplomatic communications during World War II.

Even if its counterintelligence capabilities sometimes trailed its collection technologies, Japan was never too slow off the technological mark. Despite self-imposed constraints on the military use of space, Japan deployed transponders on the satellites of civilian agencies that transmitted images for the use of military intelligence starting at least as far back as the 1980s. Today, Japanese analysts use many of the most advanced space-based image processing technologies—reportedly being able to differentiate among five pilots and a lone protocol officer standing in line on the deck of China’s aircraft carrier.14 And, like those of most other advanced nations, Japan’s intelligence community is struggling to militarize cyber capabilities to deter, if not to protect against, unwelcome intrusion.15

Failure is the third, and often the most proximate, driver of intelligence reform. Clausewitz famously acknowledged intelligence failure in On War: “Many intelligence reports in war are contradictory; even more are false, and most are uncertain.”16 Indeed, although their successes are often well hidden, intelligence communities have failed famously; Hitler’s surprise of Stalin with the treacherous Operation Barbarossa and US complacency before Japan’s attack at Pearl Harbor are just two prominent examples from World War II. Another involved Major General Charles Willoughby, who played a singularly prominent role in the postwar history of the Japanese intelligence community and who engineered “one of the most glaring failures in US military intelligence history” in Korea—both on behalf of General Douglas MacArthur.17 The failure to coordinate intelligence on the 9/11 terrorist attacks, Washington’s willful ignorance of Saddam Hussein’s abandonment of weapons of mass destruction, and the counterintelligence failures thatabetted Russian interference in the 2016 presidential election are more recent American examples.

But intelligence failure has nowhere been more plentiful or storied than in Japan. The nineteenth-century shogunate was shaken to its knees once the capabilities of Commodore Perry’s “black ships” became known; imperial militarists and pan-Asianists were unprepared for US resolve in the Asia-Pacific War; and the most famous tactical intelligence failure of the Asia-Pacific War occurred after a US signals unit located Admiral Yamamoto Isoroku near the Solomon Islands in April 1943, allowing Admiral Chester Nimitz’s pilots to ambush his plane. In the 1970s, Richard Nixon pulled the rug out from under Japan not once but twice in “shocks” that upended the global financial community and brought China in from the cold. Subsequent unannounced and unanticipated visits by a Soviet MiG-25 in 1976 and by a North Korean missile in 1998 justified immediate changes in the way Japan practiced intelligence.

These three drivers—strategic change, technological development, and failure—forced the pace of intelligence reform in Japan in much the same way that they compelled reform elsewhere. But this is not merely a story of Japan as a normal nation. We can more easily detect what is distinctive about the history and practice of intelligence in Japan when we observe how the drivers converge with the specific activities in which national intelligence communities are engaged—the “elements” of intelligence. This volume identifies and examines six such elements over time: collection, analysis, communication, protection, covert action, and oversight.

1. The White House vignette is from Andrew, Christopher M. 1995, 164. For the President’s Eyes Only: Secret Intelligence and the American Presidency from Washington to Bush. New York: Harper Perennial.  “Licensed skullduggery” is from le Carré, John. 2017. Legacy of Spies. New York: Viking.
2. Turney-High, Harry Holbert. 1971, 112. Primitive War: Its Practice and Concepts 2nd edition. Columbia: University of South Carolina Press.
3. Federation of American Scientists, ed. 1996. “The Evolution of the US Intelligence Community: An Historical Overview.” http://fas.org/irp/offdocs/int022.html.
4. Andrew 1995, 16–17. See full reference above.
5. This perspective is often associated with former defense secretary Donald Rumsfeld, but it has long been part of intelligence lore. See Thomas, Stafford T 1988, 217. “Assessing Current Intelligence Studies.” International Journal of Intelligence and CounterIntelligence 2, no. 2: 217–44.
6. Lathrop, Charles E, ed  2004, 325. The Literary Spy. New Haven: Yale University Press.
7. Laqueur, Walter. 1985, 4. A World of Secrets: The Uses and Limits of Intelligence. New York: Basic Books.
8. Deacon, Richard. 1983, 2. Kempeitai: A History of the Japanese Secret Service. New York: Beaufort Books; Elphick, Peter. 1997, 36. Far Eastern File: The Intelligence War in the Far East, 1930–1945. London: Hodder & Stoughton; Strategic Services Unit, ed. 1946, 29. “Japanese Intelligence Organizations in China.” Declassified US government document available at http://www.foia.cia.gov/sites/default/files/document_conversions/1705143/JAPANESE%20INTELLIGENCE%20ORGANIZATIONS%20IN%20CHINA%20%20%20%28WWII%29_0001.pdf.
9. China Military Online, 6 June 2017, http://english.chinamil.com.cn/.
10. Kotani Ken. 2013, 189. “A Reconstruction of Japanese Intelligence: Issues and Prospects.” Chapter 10 in Intelligence Elsewhere: Spies and Espionage outside the Anglosphere, edited by Philip J Davies and Kristian C Gustafson. Washington, DC: Georgetown University Press. Frustration with the lack of community was expressed more often in the interviews conducted for this book than any other single point.
11. Kuroi Buntarō 2005a, 236. "Wārudo Interijensu [Worldwide Intelligence].” Gunji Kenkyū (September): np.
12. See interview with Magosaki Ukeru in Kuroi, 2008, 261. Interijensu no Gokui! [The Mysteries of Intelligence]. Tokyo: Takarajimasha.
13. Kōno Tarō, Mabuchi Sumio, and Yamauchi Kōichi. 2013, 94-95. “Nihongata ‘Supai Soshiki’ no Tsukurikata [How to Build a Japanese-Style Spy Organization].” Chūō Kōron (May).
14. Interview, former Ministry of Defense official, 29 November 2017, Tokyo.
15. The National Defense Program Guidelines issued in December 2018 emphasized both offensive and defense cyber capabilities.
16. Quoted in Warner, Michael. 2014, 19. The Rise and Fall of Intelligence: An International Security History. Washington, DC: Georgetown University Press.
17. “Glaring failure” is from Haynes, Justin M  2009, iv. "Intelligence Failure in Korea: Major General Charles A Willoughby’s Role in the United Nations Command’s Defeat in November 1950.” MA thesis presented to the US Army Command and General Staff College, Fort Leavenworth, KS.