Using 'Big Data' And AI To Understand The Patterns Of Our History And Tell Us About Our Future

One of the driving forces behind much of my work over the past quarter century has been how we can use massive datasets and computing platforms to help us understand global society, from the patterns that underlie our behavior to the narratives and emotions that make us human. From cataloging our past and visualizing our history to finding the patterns of history and weaving all of those narratives and stories together, I’ve long been fascinated with what becomes possible when our digital and digitized history is coupled with massive computing power and directed at the grand challenge questions around who we are, where we’ve been and where we’re heading. What might the future of data-driven exploration of humanity hold?

Perhaps the greatest contribution of the web era has been the speed with which it has ushered in the digital revolution. By creating a medium and business model for the production and curation of born-digital content at global scale, the web has reshaped the way we think about the production and consumption of information, encouraging everyone from businesses and governments down to ordinary citizens to publish their thoughts, beliefs, emotions, ideas, information and experiences to the world. The web collects all of this, serving as a single access point to society’s present thoughts and past knowledge. Simultaneously, the consumer orientation of the web and ease of publication means our past is increasingly being digitized into computerized existence.

Simultaneous to this data revolution, the cloud revolution that has underpinned the web’s rise has transformed the economies of scale in storing and processing all of this data, giving us access to the computing environments and tools necessary to extract patterns from petabytes.