The Deep Learning Revolution — The Shock of 2012

Last time, we witnessed two winters. Expectations deflating, researchers leaving, the neural network pushed to the margins of research—a long story of stagnation. Yet beneath the frozen ground, three seeds were quietly sprouting as they waited for spring: vast data, computing power of a wholly different order, and the few researchers who had endured the winter. In 2012, when those three met at a single point, the world was, quite literally, transformed.

The Hard Problem of Not Being Able to Read a Single Image

For a human, telling whether the thing in front of you is a cat or a dog is an instant feat that requires no thought at all. But for a computer, this was long a formidable problem. What the machine sees is nothing but an endless string of numbers, and extracting the meaning “this is a cat” from it is harder than one might imagine. Under the old approach, humans had to hand-teach features one by one—“the shape of the ears,” “whiskers”—and such methods quickly broke down in the face of real-world variety.

What set the stage for the turning point was one enormous body of “teaching material.” Released in 2009, ImageNet was a database of unprecedented scale: millions of images, each hand-labeled with what it depicted. Using it, research teams around the world began competing in an annual contest over “how accurately can you classify images?” For a long time, the accuracy rate crept up only by inches. Cutting the error rate by even a few percent was a major achievement—such was the conventional wisdom of the field.

2012: The Day Conventional Wisdom Was Turned Upside Down

At that 2012 contest, an upset occurred.

Geoffrey Hinton of the University of Toronto and his students, Alex Krizhevsky and Ilya Sutskever, entered a program that would come to be called AlexNet. It was the very thing long dismissed as outdated: a deep neural network. The result was stunning. AlexNet’s top-five error rate was about 15 percent—leaving the runner-up’s roughly 26 percent far behind by an enormous margin.

In a world where competitors had been fighting to shave off a few percentage points, a gap of nearly ten points suddenly appeared. This was less an improvement than a rupture. The researchers in the room watched their conventional wisdom collapse before their eyes. The very method abandoned during the winter had just identified objects more accurately than anyone else—a fact presented in a form that could not be denied. The long-running cycle of spring and winter met its decisive end here.

The Winning Edge Was a Gaming Chip

So why did the old method suddenly revive? The key was an unexpected tool—the GPU (a graphics-processing semiconductor), originally built to render video and game screens.

Training a deep neural network requires performing a great many similar, simple calculations all at once. This was work better suited to a GPU, which handles simple tasks in parallel en masse, than to the conventional processor (CPU), which excels at working through one complex operation at a time. Krizhevsky and his colleagues used these gaming chips to dramatically speed up the network’s training, completing within reach a scale of computation that had until then been impractical.

Here, the three seeds finally met at a single point: the vast data to learn from (ImageNet), the computing power to process it quickly (the GPU), and the researchers who had endured the winter and kept refining the method. Had any one of them been missing, the revolution would not have happened. The “semiconductor born from sand” we saw in the previous series reappears here as the heart that drives intelligence. Intelligence as a resource can only stand once it rests on the foundation of computing power.

The Talent War Among America’s Tech Giants

In 2013, with the shock still fresh, a symbolic event occurred. Google acquired the small company Hinton had just launched with his students. The aim was less the technology itself than the minds that produced it—the researchers. After the acquisition, Hinton joined Google and turned to applications such as image and speech recognition.

This was a sign of the storm to come. Having witnessed the shock of AlexNet, America’s IT giants began scrambling to gather deep learning researchers. Top minds at universities moved to corporate labs, drawn by extraordinary compensation. Search, advertising, images, translation—the giants were unwilling to let go of a technology that could remake their services wholesale. And so the frontier of AI shifted its center of gravity from universities to companies, becoming a race that only those with abundant funding and data could run.

That said, accounts of how this talent was valued and treated vary in tone across the testimony of those involved and the press coverage, and the full picture cannot be told as a single, seamless narrative. Even so, the fact that AI was transformed after 2012 from a “laboratory dream” into “the main battlefield of big business” was plain for all to see.

The technology thus blossomed all at once in America. But it was not only Silicon Valley’s companies that watched this revolution closely. Across the Pacific, one nation had begun to see AI not as mere business but as a strategy that could decide the fate of the country. Next time, the stage shifts to China—a story of how a single game of Go awakened a great power.