Tweet

The Challenge is Staying Afloat in a Rising Sea of Data

Guest post by: Andy Marken

Article Overview: Getting the feeling you just can't keep up with the information, ideas, data, images that are coming at you on your screen every day? It's true...suddenly we're way past petabytes of data. It's so yesterday! If only our minds could grab it, file it in appropriate areas and bring it back just as we need it. Maybe someday when singularity becomes an everyday thing - look it up in Wikipedia and then think about tomorrow. Don't sit back and think too long thought because there's more information piling up ready for you to assimilate, dissect and use. We're becoming a world of knowledge workers so work.

The Challenge is Staying Afloat in a Rising Sea of Data

"A $40 million computer tells you you're chasing an earthquake, but you don't believe." - Capt Mancuso, "The Hunt for Red October," Paramount Pictures (1990)

One of the toughest jobs in any country, any company is gathering, hoarding, protecting information that may give them an edge against "them."

And we must be doing a helluva' job because organizations and individuals are accumulating data and meta data at an ever-increasing rate.

The sheer volume that's coming at us is causing us to experience information overload...emails to answer, virtual friends to contact, YouTube videos to watch, games to play, movies to sneak, reports to read/digest, stuff you have to find/do in the virtual world...just in case.

Computers, smartphones, tablets are helping to accelerate our hunger for more data and our ability to create new/better information.

Does it seem as though there's so much information, you can't keep up with it? Can't make sense of it?

You're right!

As Jack Ryan said, "Welcome to the New World, Captain."

The Next Big Number

EMC (a world leader in big storage stuff) recently sponsored a report by IDC (international research firm) that says we have finally cracked the Zettabyte barrier.

Piling Up - When it comes to storage capacity a MB isn't much, a GB is barely tolerable and a TB is almost a necessity for families today. It wasn't that long ago that we considered a PB as an amazingly big number; but today, we talk about a Zettabyte without blinking an eye. Storage requirements are more than doubling every year and there's no end in sight -- as long as we can properly manage it and find the specific information we want and need. Chart - IDC

This year, we will have created, replicated more than 1.8 Zettabytes of data (images, video, music, graphs, stuff) which represents a 9x growth in just five years.

And we're getting better at it.

Over the next 10 years, data centers/repositories - enterprise, cloud, personal - will grow by a factor of 50 while the number of files these locations will have to deal with will increase by a factor of 75.

As Admiral Painter commented, "This business will get out of control.'

Why are we capturing, storing, hoarding all of this data?

Industries of Knowledge

The global economy isn't based on "things" anymore; it has become a knowledge economy and the countries/organizations with the smartest, best-trained workforces, best ideas and the ecosystem to support innovation will be the leaders.

The value of the data you and everyone captures was expressed by Stewart Brand, founder of Whole Earth Catalog, at the first Hackers' Conference in a conversation with Steve Wozniak (co-founder of Apple) in 1984.

In his discussion with Woz, Brand said "On the one hand, information wants to be expensive, because it's so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free because the cost of getting it out is getting lower and lower all the time. So you have these two fighting against each other."

Viola!! Brand gave a whole band of Black Hatters (hackers) a rallying cry.

No good deed goes unpunished.

The innocent conversation also gave bands of super-bad, super-annoying Black Hatters like Wikileaks, Anonymous, Lulz Sec and others a rationalization for their activities.

But simply outing private data because you can doesn't accomplish anything. In fact, most of the time, it does more harm than good.

As Capt. Mancuso observed, "The hard part about playing chicken is knowin' when to flinch."

Opening Data Vaults

Whether they are being pulled or pushed, many organizations are opening their data vaults to countries, companies and individuals to develop even better information, better solutions.

The most valuable commodity companies and individuals have is the data they want to save as securely as possible. With knowledge becoming such a valuable commodity, some organizations have found that the more open they are with their information, the more solutions they find themselves. Art - NYTimes

The World Bank, long a treasure trove of information, is making that information available to democratize development economics.

Historically, the bank hoarded information and released bits and pieces in a choreographed fashion to a specific audience.

With the change, organizations and people everywhere have been tapping into and using the free data.

The information has been used to reshape government policies around the globe; and in some instances, to reshape governments when people use it with their smartphones, Twitter statements, Facebook/Google+ postings.

The new computing/communications tools create - you guessed it - even more data.

Graphic Illustration - There are almost limitless ways to describe how our data is increasing. By 2020, the digital universe will be 44 times larger than last year and will include the Petabytes of storage individuals will have in their homes. Source - IDC

With the right tools, IDC points out that the overload of information is becoming increasingly valuable and available across all sectors of the digital universe.

Improved global network connectivity, lower cost computing power and streamlined/focused apps make it easy for companies - large and small - to tap into and use transactional data, warehoused data, metadata and data located virtually anywhere to respond to, assist and serve customers.

Created, Consumed - While producing data from data, about data, may seem like a whirlpool or black hole; it is our ability to analyze all of the data surrounding other data that delivers answers to unasked questions. From tomorrow's infrastructure, apps, governance and consumption, big data is also changing the datacenter and its team. New opportunities appear with each new content delivery source that is launched. Source - IDC sponsored by EMC

The big data that is getting bigger all the time isn't just original content waiting to be used.

Improving with Age

It's more valuable than ever because data is captured about the content consumers use, which is arguably even more valuable than the original content.

In other words, the data around the data.

As Jack Ryan said, "It is wise to study the ways of one's adversary."

Better Information, Better Assistance - Companies, both large and small, are increasingly turning to the broad range of internal/external information to manage their organizations, plan products/services for tomorrow, and speed/improve product and customer support.

As IDC explains, big data calls for new technologies to economically extract value from huge volumes of a wide variety of data so that it can be captured, analyzed and used.

Fortunately, while the mass of information is growing exponentially, the cost of storing it is steadily dropping.

Decreasing Data Storage Costs

Demand Increases, Costs Decrease - As companies and their IT activities change, there will be changes as the digital universe expands. It will be more complex, require more processing, more storage and increased management which will raise the levels of investment. Fortunately, storage costs will decrease to meet the exploding storage requirements. Source - IDC sponsored by EMC

That's a good thing, at least in our household where we're constantly adding TB drives.

Why?

Because even in a free information society, there are some data that must be safely/securely protected.

Privacy Still Vital

In the digital universe, there are key areas of security that must be maintained:

- Privacy - say your email address on a Sony account or YouTube upload

- Compliance - email retention in the event of litigation or legal compliance

- Account information - such as recent customer/user identity thefts

- Confidential - customer lists, trade secrets

- Lockdown - the highest security for medical records, financial transactions, military actions,

As Jeffrey Pelt pointed out, "I'm a politician which means I'm a cheat and a liar, and when I'm not kissing babies, I'm stealing their lollipops. But it also means I keep my options open."

The bad Black Hats argue that all data, all information should be freely available; but somehow we don't believe that will hold up in any court...including the court of public opinion.

Our desire to accumulate more and more information is insatiable. The key to your success though is how you filter, discard and use that information ... and how often you're right or wrong.

There's still hope, as long as we review the data and rely on gut feelings.

As Jack Ryan said, "I had a 50/50 chance. I needed a break."

Finding the data in an ocean of files is becoming a business opportunity for app developers and submarine drivers.

As Rich Wallace, publisher/editor of The Next Silicon Valley pointed out recently, there are increasing opportunities for entrepreneurs, start-ups and established firms everywhere to take advantage of today's knowledge-based economy.

Related Articles
  Harvard Business Review Hit and Then Missed the Mark on Sales
  Staying Afloat via cash flow financing – Cash Flow for Business Solutions
  Data Loss - The Scariest Term in Business Today
  Going North when the Economy is still South!
  How to Deal With Rising Costs at Your Small Business
  How to maintain data privacy?
  Inspirational and Motivational Tips for Giving Early Rising a Chance
  5 Things Business Owners Must Consider During Growth or Survival Periods
  Recover Hard Drive Files Now
  Too Many Sales Reps Are Wimps
  Becoming the Master of Customer Data
  Do You Have What It Takes To Start A Data Entry Business?
  Legitimate Work at Home Data Entry Jobs: Dont Look Too Far!
  Do you want to succeed in a down economy? Many have; you can too.
  What Does Expensive IT Really Look Like?
  Data processing Requirements
  How Important is Staying Power in your home based online business?
  Watch Your Business Expenses (Self-Employed)
  The data storage prayer
  What are the 3 biggest challenges faced by supply chain/purchasing professionals today? (Survey Result 19)

Article Tags: data images, helluva job, information overload, jack ryan, mancuso, meta data, number br, paramount pictures, red october, repositories, sheer volume, storage capacity, storage requirements, video music, virtual friends, virtual world, welcome to the new world, world leader, zettabyte, zettabytes



Related Forum Posts
Re: Job market slump is a guy thing - Yes, it does complicate things when you get promoted and then manage your previous peers. Rising through the ranks can really "urk" the people who aren't rising with you, especially if they have more of a negative view. You can then be the subject of their frustration. The important thing is to realize this is about them and not about you. Otherwise, it can be really difficult as a leader.
Re: What is your biggest challenge? Today? - Staying healthy is important as I get very little done when I have the flu. I basically shut down for a week to recover properly and continue to work hard.
Re: The Top 50 Productivity Blogs To Watch in 2010 - Thanks for that list Evan. I have bookmarked it to go through more carefully when I have some spare time. What you say here is so true, the biggest problem with working on the internet is that it is so easy to get side tracked. Staying focused is really the hard part. MichelleJ
Re: Does Your Website Have a Robots.txt File? - HI Carol, Your robot file is correct; no lets work on your Page Titles & Meta Data. Do you have access to create different page titles for each page of your website? Do you have your FTP information...please do not post it here but just let me know if you have access. Jeff
Re: What I Enjoyed Reading This Week - June 11 - Hi Evan, Thank you for the reading material. I look forward to the general release later this month as outlined in the 'Google Analytics To Add Search Query Data From Webmaster Tools' article. Anything that helps us to understand maximise extra data from Google Analytics can only be of benefit. regards, Mal.


Share this article with your friends. Fund someone's dream.

Leave a comment below or share on the left and you'll help support entrepreneurs in Africa through our partnership with Kiva. Over $50,000 raised and counting - Please keep sharing! Learn more.

Newsletter

Get advice & tips from famous business
owners, new articles by entrepreneur
experts, my latest website updates, &
special sneak peaks at what's to come!

Name:
Email:

Popular Articles

Tips to Take Control of Credit Card Debt

Ready for a Fresh Image?

Top Ten Feeble Follow-up Attempts to Fathom

Suggestions

Email us your ideas on how to make our
website more valuable! Thank you Sharon
from Toronto Salsa Lessons / Classes for
your suggestions to make the newsletter
look like the website and profile younger
entrepreneurs like Jennifer Lopez.