Use big data wisely and it can provide big insights

Robert Kirkpatrick of UN Global Pulse in SciDev.net: The variety of data generated every day — from satellite images to people’s geotagged photos of their breakfast — is astonishing. The sheer quantity now being produced is hard to fathom: experts estimate that 90 per cent of all the digital data in the world has been generated over the last two years.

The term ‘big data’ refers to the information generated by the everyday use of digital services, produced so frequently, in such diverse ways and at scales so large that it cannot be stored or analysed through traditional means. Analysis of this information can provide valuable insights about human behaviour and intentions.

This new era of big data represents significant challenges and opportunities for global development. With support and guidance, the development community can embrace the craft of ‘data science’ to enhance evidence-based decision-making. Yet it is important that humanitarian and development practitioners and policymakers understand that data on its own can’t inform the development agenda without a detailed understanding of its local context.

Global Pulse, a UN innovation initiative established by secretary-general Ban Ki-moon in 2009, is working to build development sector capacity to make use of new sources of digital data while understanding its local context.

The initiative is exploring how powerful the insights from big data can be. One example is our research project looking at how Twitter can provide information on how people are affected by food price spikes. We analysed millions of public tweets from between 2010 and 2011 for particular key words. In one data set containing thousands of tweets from Indonesia discussing the price of rice, we saw that the volume of tweets about staple foods correlated closely with actual spikes in their cost. Learning how to identify and understand such data could help the development community anticipate trends....

Image of a data cloud by Richard Cyganiak and Anja Jentzsch, Wikimedia Commons, under the Creative Commons Attribution-Share Alike 3.0 Unported license