Data is a precious thing because they last longer than systems – Tim Barnes Lee
Introduction
Big Data, as the name implies, is a collection of huge amounts of data. Data has always been important to mankind and society by various means. But in the past few years data has been collected and used for a variety of purposes. The major factor that differs Big Data from normal data in a DBMS or RDBMS is that Big Data is very large volume, it has no defined structure, it includes a lot of data that is being created from various channels like emails or social media like Twitter, Facebook, LinkedIn and so on.
What can be done with a lot of data?
When we have gathered the data, the best use of the data can be made in the following three ways:
- To understand what has happened
- To understand and explain why it actually happened
- Most importantly predict what will happen
Big Data has always been around but this term has become more popular recently. Think of a weather forecast system that can predict weather based on data collected. Another great example is a traffic support system of a GPS, that is reading data from various satellites to re-direct you to the roads with less traffic or highways and so on. If you dig into the details of how these systems are being made and trusted it is fully based on their capabilities of acquiring, collecting, processing and then doing a predictive analysis with that data to produce results that are almost accurate.
For the past few years forecasting and predictive analysis with data was done by National Weather Labs, the intelligence community and so on but now it is much more since then.
How Big Data is being captured and who contributes to Big Data
Big Data has grown tremendously and it's increasing like anything in an uncontrollable fashion. Of most of the world's data today, approximately 90% was generated in the last two years. The reason of that is our day to day contribution in various digital forms of data, for example Facebook, Tweets, pictures and so on.
Hence, it won't be surprising that in fact the web does millions of activities per second and so huge peices of data is being stored in server logs. You must have heard that Goggle stores (and some people say steals) all the information of your activities. For example, your search strings. This is the same with any search engine.
One of the most interesting questions is who actually contributes to this huge data-building process. Well the answer is simple, we do. Yes, we all are helping the various companies to gather, process and then predict via the data that is being contributed on a day to day basis by approximately over 800 million active Facebook users, approximately over 40 billion photos, over 200 million active users on Twitter and so on. Just FYI; Facebook's Big data analytics tool can process more than 20 billion events a day.
So, whatever you do actually adds into the avalanche of data for further processing to serve mankind with the information they need and most importantly at the right time when they would really need it (an example will follow in the article below).
Two Great Examples of where Big Data has changed the world
India
You must not have forgotten Amir Khan's Stayamev Jayate, one of the most popular shows released in May 2012, was using Big Data. During the execution of the show 1000+ engineers were working behind the scenes to capture all the data that will be created on Facebook, Twitter, websites and blogs and so on to actually capture not only the opinion about the show, but the category of people, from where including count of positive and negatives and so on.
The producers utilized the technology to define the future episodes, the channel reported traffic of 40,000+ Tweets during the 90 minutes of the show.
USA
Obama the President of USA, was depending on Big Data for his election campaign. His app “Obama for America” connected all the people who were in favor of him and joined the campaign. Over 2 million joined the app and it shows the people in your neighborhood that like Obama, their name, age, address and gender and so on including various other personal details.
How Big Data is being used today and what is next
Based on your search in Bing and Google, you will observe some advertisements on your social media sites like Facebook that are related to your searches. Companies are working on an interesting path that not only utilizes Big Data but also combines various technologies together. Assume, in the near future you walked into a store and, based on your last purchases made in the store, you begin to get information about the deals, discounts and so on on your phone, Twitter and so on.
To make this possible, the store needs to grab all the data including your purchases, your Facebook likes, your searches on various search engines and so on and as soon as you enter into the store they sense you buy your handheld device and start analyzing and processing the data to predict what you might like based on the patterns of your purchase and search history. Interesting isn't it?