Big Data Questions for the Layman

• 90% of data was created in the last two years. Why is that? 
• How does that affect security professionals? 
• How does that affect the everyman?

According to various sources 90% of the data in the world was created in the last two years.  The main reasons are: the rising popularity of social networks, our willingness to share more and the type of information we create/consume/share.  Let’s have a look at each of these in turn.

Rising popularity of social networks

Facebook, for example, grew from 197 million to 901 million active users within the space of 3 years (source: jeffbullas.com).  Google+ figures tell a similar story, within a year its user base grew from less than 50 million to 500 million (source: farotech.com).  These are the two behemoths of social networking; however the figures from other networks, such as LinkedIn, Twitter, etc. suggest a similar story.

Willingness to share

And, this growth has made all of us much more comfortable using the networks, connecting with family and friends, posting photos, videos, songs, etc.  

In a sense, this is similar to the e-commerce trends. In the early 2000s only a small number of people were transacting over the Internet and the security, privacy and other concerns were quite high, and justified.  These days hardly anybody pays attention how the transaction occurs.  What is important is the result – a purchase made at an acceptable price with almost 100% guarantee of delivery.

Mark Zuckerberg even speculated that the amount of information we share doubles every two years.  

Type of information created/consumed/shared

Until recently creating and posting “rich media” for others to see and use required a certain amount of technical skills.  At present, with a large percentage of population (50.4% in US, according to Nielsen) carrying smartphones in their pockets/handbags, it’s never been easier to create a 3-min video of little Johnny playing with his sister. And, the size of this video on iPhone 4 would be 240Mb, while on iPhone 4S or 5 that figure would go to 540MB. 

Even with compression, let’s say when uploading to Facebook,  the data amounts are staggering compared to a spreadsheet considered “state of the art” only a few years ago (2005 seems somewhat pre-historic these days in “Internet chronology”). 

This type of data explosion is certain to affect a growing number of professions.  And information governance and security professionals, will be amongst to first that will have to adapt and deal with this new reality.  Organisations are already starting to recognise the importance of Big Data and how it can deliver competitive advantages.  As a consequence, this realisation will further accelarate the drive to start using these technologies.  

Thus, the question in the mind of many executives will become “how to do it”.  It’s certain, existing systems and infrastructure are not very well suited for Big Data purposes.  Some of the more obvious options would be:

• develop expertise in-house and introduce new infrastructure;
• outsource Big Data processing to companies specialising in this area;
• a mixture of the above;
• or a new model, e.g. utilising Platform-As-A-Service (PaaS), Software-As-A-Service (SaaS) or something currently being dreamed/prototyped in a garage somewhere in the world

Any of these scenarios imply a certain amount of change.  And, how this change is decided upon, implemented and then managed will be of concern to governance and security professionals.  Why? They will need to provide advice, help other units within their organisation and monitor the new processes afterwards.

This is a tide which is already irreversible.  Organisations can’t afford not to deal with Big Data because their external stakeholders have already embraced it.  Our lives have been enriched by new technologies and we will continue to share lots of photos and videos.  We will ask for sensors to be attached to us so that we can monitor our blood pressure, or how well we exercise or sleep.  And, because we know that, through various sensors, a trained professional can be monitoring our parents’ health, living on their own, 24 hours. 

Big Data has made big promises and now the general population is keen on those being delivered.