Detail from a data map showing deaths from cholera in London in the 1840s—an early example of “big data” collection

It’s a glorious Saturday morning, a day for kayaking or sitting in a sunny courtyard with a coffee and the silly bits of the FT. But here I am, stuck in a bunker at the British Library with a bunch of Generation Y geeks telling me that my nice, tidy world of hypothesis, experiment and knowledge generation is about to end.

I am attending the Science Online conference, in which the usual scientific lexicon of sample size calculations, placebo-controlled trials and statistical significance tests is nowhere to be seen. The talk is of scraping and mining, terabytes and petabytes, of algorithms. It’s the language of Big Data—the ocean of information being generated by ever-larger telescopes, ever-cheaper genetic sequencing techniques and ever more Facebook users. As Royal Society president Martin Rees has written (Prospect, November 2010), Big Data will allow us to mine and mash our way to unexpected discoveries and insights. It allows us to ask new questions, ones that we couldn’t have asked when science depended on the work of a few people in a single lab working in a limited area of knowledge with just a few gigabytes of processing power. Some people say that Big Data also changes the way that we ask questions. Gone are the days of hypothesis-driven science as we know it. Nowadays, it’s all about pattern recognition.

Click below to access the full atricle
Prospect Magazine