Why Does Speculation Persist in the Age of Predictive Data?

We like to tell ourselves a certain story about the development and advancement of modern science. It goes something like this: early humans lacked the methods or means of rigorously testing the world around them and assessing outcomes in order to predict future events. Therefore, they relied on divination, augury, and prophecy to foretell what tomorrow holds.

The scientific revolution was supposed to have changed all of that. Our modern world is built on empirically verified laws, and when we want to know what will happen next in any given domain, we have stable, agreed-upon methods for examining evidence and drawing conclusions with a high degree of predictive certainty. We’ll never have complete confidence about the future, but with the accumulation of knowledge and the collection of data over time, we know vastly more than both our distant ancestors and our grandparents. We’re so sure of our predictive data now that we use it everywhere: social media algorithms are built on it; marketing campaigns thrive on it; climate change policy is authored by many scientists who preach the value of understanding it. Even professional sports teams use masses of performance data to assess their players and predict future outcomes.

Which leads to a question: why do we still speculate so much in the present? The past was full of speculation, and the history of speculation itself is bound up with prophecy, gambling, lotteries, and other activities. Speculation is flimsy, ethereal, and dreamy: we still say “I don’t want to speculate,” and we speak of “idle speculation” when dismissing it. It seems like an admittedly insubstantial way of envisioning the future, much like “speculative fiction” itself is a projection of a dream-world that we know does not exist.  

What we call “speculation” in finance hardly fares any better. Though it is largely condemned as greedy and often parasitic on moral grounds, that condemnation actually shares a scientific root too: to speculate is to gamble on poor evidence about the future. To invest or to budget or plan, on the other hand, is to make sound financial choices based on wise readings and methods, having studied all the available data, assessed one’s risk tolerance, and prudently considered the consequences for the future. Speculators, instead, recklessly and irrationally pour money into silly things like meme stocks or NFTs--so the story goes--without seriously analyzing the data.

There are many potential answers to the question of why we speculate in an age of science and Big Data, and they range from behavioral to psychological to practical. But what is perhaps most astounding is an irony underlying it all: our widespread faith in modern scientific methods, combined with the ready availability of troves of predictive data, has not lessened speculative habits. In fact, we only speculate more, and we even use computers to speculate for us, training them with algorithms to become financial speculators themselves now. Millions of flash trades are executed now based purely upon computational judgments that are speculative both in the conceptual and, in plenty of cases, financial senses. On one reading, it’s precisely because the financial future is so much more knowable and predictable that such risks can be taken and trusted to computers rather than to human judgment.

Either way, the modern data revolution--however we want to characterize it--has dramatically reshaped how we speculate. It makes sense, then, that the history of moral, legal, and financial regulation is full of attempts to push speculation into zones of vice and illicit behavior. These efforts took off especially once the ticker machine, invented in the 1860s, became a prominent means of speculating for both professional investors and commoners. In the following decades, speculation had to be heavily policed, the logic went, in order to protect society at large from the collateral dangers of a few poor, untrained, and wild interpreters of data. Or, at worst, to cordon off their activities as gambling at their own risk. 

But as we know, speculation was and remains quite central to what happens every day on Wall Street, just as it was during the financial revolution in England in the early eighteenth century, or in the commodity markets of Chicago in the mid-to-late nineteenth century. Speculation was never relegated to back alleys, it never went away, and is never going away, just as its conceptual sense--to contemplate the future--is a key part of how we imagine our everyday lives. For that reason, the Supreme Court had to take up a landmark case in 1905 whose bearings are still important for how we understand rational planning, investing, and speculation today. 

Chief Justice Oliver Wendell Holmes framed the issue in his majority opinion in Board of Trade of Chicago v. Christie Grain & Stock Co., which concerned commodity speculation, but was also set against the context of rampant semi-policed gambling dens known as “bucket shops.” Holmes argued that “people will endeavor to forecast the future, and to make agreements according to their prophecy. Speculation of this kind by competent men is the self-adjustment of society to the probable.” Holmes reaches back to the language of “forecast[ing]” and “prophecy” to characterize speculation, acknowledging that it is, at its core, a game of primitivist guesswork. But he qualifies his claim in several key ways.

First, he installs “competent men” as the guardians of speculation. It’s safe, he implies, when it’s done by wise, well-studied, level-headed males--because they are educated, know how to read financial data and turn it into rational projections. It’s worth noting that the first woman-owned brokerages had just opened on Wall Street a few decades prior, with great success, and that women speculators (most notably Hetty Green) were increasingly making headlines and making waves in America in this moment too. They were largely seen as acting only on their passions and whims, and Holmes’s conservative defense of speculation aims to stave them off. 

And second, Holmes calls speculation a “self-adjustment” by society as a whole to what is “probable”--that is, to what is likely to happen. He naturalizes speculation as a process, something internal and integral, a self-regulation that, he later explains, stabilizes prices and equalizes outcomes in the end. Here, data magically turn into evidence, and thus into a self-fulfilling prophecy. This view of “speculation” hardly sounds like the frenzied gambling we’re used to imagining, and for good reason: Holmes was attempting to legalize a wide swath of activities under the banner of a broad concept, all while still making illegal a host of other practices that, in his reading, unnecessarily and arbitrarily introduced elements of risk. He drew a line, but even that line was blurred within a few years, and by now it’s as hard to distinguish as the gray area between games of skill and games of chance.

Which brings us back to the central question at hand. During this pandemic, all of us have studied many data points to assess our risks and predict how safe our futures will be under X or Y scenarios. Even when there has been no shortage of data, even when the data have overwhelmed us, the future has never been made certain and clear for us by them. Instead, we have had to become speculators to some degree. Not unruly gamblers, but projectors uncertain of the future despite all the data around us. We speculate because even all the advances of modern science, and all the collections of data we can store are--to borrow Mary Trump’s wonderfully descriptive phrase--both “too much and never enough.”