BIG data: its gains, losses and absurdities

In my first post ever on this blog I talked about the something called the 'knowledge economy'. This is one of the most common memes of our time (if you have no idea what a 'meme' is, click here). Like other things in this world, memes tend to cluster around other memes and one of them as far as 'knowledge economy' goes tends to be something called 'big data'.

Big data is just what it is; enormous amounts of raw, unprocessed information that cannot be sorted or dealt with using traditional, hands on means. No human can look at big data and derive any real meaning from it. Big data is usually handled with the help of computers. Examples of big data being used today abound in the world (meteorological data being used to forecast the weather; the richer the data set, the more accurate the forecast) and the problem has also given birth to many companies, the most commonly cited being Google (whose very name is derived the term 'googolplex' which is 10^10^100). The company's computers crawl through the web, indexing trillions of pages to make them searchable for all of us. Despite that, the internet remains far from being entirely indexed for 2 reasons; primo its growing everyday and secundo its so damn big. As a side note, check out this nifty website to see how big the internet probably is (I say probably because the map is obviously incomplete).

Big data exists the moment something exists in this universe. The universe and all its contents has existed for 13.7 billion years. The trick is capturing that data, storing and analyzing it for useful patterns. The tools for performing these actions have only arrived in the last two centuries in the form of cheap, compact storage devices, powerful processors, networking technologies and, most importantly perhaps, necessity.

To illustrate the importance of necessity's role in spurring the development of tools for handling big data, take a look at this TEDEd video:


Since the time when IT first became a ubiquitous part of our lives, big data has become relatively easy to collect. Mapping has enjoyed the fruits of big data; the maps of today are no longer physical pieces of paper that my parents and I used extensively on our excursions around the city of London in 2002. Nowadays people have access to compact, electronic maps on their smartphones which are rich in social data, never become outdated and can even feature live updates on such important things like traffic information (the company Waze, which was purchased by Google in June 2013. allows you to access social info on traffic with GPS enabled phones).

However, what good technology has given us can also be used against us. Politics aside, big data technology has also allowed security agencies to look into our activities with unprecedented impunity. Privacy policies and laws may forever be left in the dust as the pace of technological development outstrips the ability of lawmakers to protect our digital privacy online (if you believe in such a thing anyway). Though the aim is purely (I think) to catch the bad guy, I guess we are in real danger of being wrongly accused by overzealous/over-legislated security agencies or caught by nervous, tyrannical regimes seeking to protect their illegitimate hold on power. All thanks to big data and the technologies it has spawned.

But the ultimate problem of big data (especially of the social media variety) is its terrible need to be verified or curated (and that's why crowd-sourced projects like Wikipedia still needs editors or there would be chaos) to ensure we don't end up turning noise into conclusions. Taking unverified data as real information is the biggest absurdity of the internet today. Perhaps that is the reason why teachers hate students referring to Wikipedia for information. Though I disagree with not using Wikipedia completely, CITING it in presentations and essays is another thing entirely. Its just not there yet. Give the technology time to mature and maybe...

Some also say that social media (think Twitter and Facebook which also deal with big data produced by millions of narcissistic humans) could be the new source for ALL news. While that is true if you have friends who love tweeting the news or giving accurate tweets (you could end up being given a skewed picture of the news), you really should be able to verify that raw piece of data with multiple sources. Big data is nothing if there is no useful way of deriving valuable and realistic knowledge from it, that is if there is any useful thing to be derived from it at all, which is another problem entirely.

The one thing that I find most amazing about big data is how much we humans have produced in our existence; more than 90% of it has been produced only in the last several years. By the time my grand children are born, we're going to have a real problem of storing that data let alone finding ways of scrutinizing it for useful patterns. But as usual, necessity will always mother an invention for the job.

The Best Analogy for the 'God Particle'

Last year there had been great excitement within the circles of theoretical physicists as well as particle physicists concerning the possibility that the  Large Hadron Collider, the world's biggest and most powerful particle accelerator, might have given scientists enough data to clinch the formal discovery of something called the Higgs Boson particle, otherwise known as the god particle (I always get goosebumps when I say that). Recent reviews performed on the data collected so far seem to indicate without a doubt that the god particle is real.

Although this is being described as probably the most sensational discovery yet in modern physics, many of us mere mortals are still unable to understand what's gotten the physics nerds at CERN (the European Organisation for Nuclear Research) all pumped up with excitement. I'm sure if it weren't for the albeit scary nickname we wouldn't be talking much about this particle in popular channels. Here's a quick summary about it:

Simply put, the god particle is so named because it is theorized that all the particles in this universe that have mass (like the protons and neutrons in atomic nuclei) all owe their massive existence to this one particle when the universe formed in a hot big bang almost 14 billion years ago. It was first theorised by a man named Peter Higgs in 1964 as part of a solution for the standard model of the universe but we needed something like the Large Hadron Collider to observe this particle in the flesh.

As important as it is however, one can't seem to find a good explanation out there about how on earth can one measly particle do all that it is theorised to do. This recent video from TEDEd provides the best analogy that I have seen yet that tries to do that in simple, layman's terms. Enjoy!

Agriculture and Climate change: the *other* Inconvenient Truth

Last week I watched a very interesting video talk the TEDEd youtube channel given by a rather relatively obscure chap (in the popular mind at least) called Jonathan Foley. He is the director of the Institute on the Environment at the University of Minnesota in St. Paul, MN, USA and in this video he talks about the issue of agriculture and its effects on the environment and climate change. The title of the talk is taken from the documentary An Inconvenient Truth which features former US vice president Al Gore talking about climate change.

I believe Foley's talk is very important for 2 reasons. First, in this highly misinformed age of ours, the media tends to focus on the hype of climate change which is unfortunate because it is such a technical problem (yes deniers, I'm talking to you) that the populace fails to follow the arguments and research and consequently ends up disputing findings because they applied common sense where one shouldn't, scientifically speaking. Climatology is a complicated thing and not something that should be abused by naysayers.

The second reason occurs as a result of the first; if you want to talk with the populace about climate change, start with the fundamentals. In this case it is the environment and agricultural practices which affects and is in turn affected by the environment. No climate change apparent at first but it is still there, its just being explained in the context of familiar human problems.

Here's the video. Enjoy!