Friday, November 1, 2013

unCon 2013 Session: Overcoming the Challenges of Making Big Data Useful

Moderator: Ola Mayer, Attunity

Today we led a great discussion at MassTLC unConference on the challenges that organizations have around moving Big Data to stay competitive and gain business insights. As always, in typical unConference style, the discussion evolved into many interesting directions including talking about what Hadoop is, data co-location, storing aging data and how to architect your datacenter information repositories in the face of large amounts of data and the need for timely analytical insights. During the session, we had approximately 20 participants with different backgrounds and field expertise including medical and hospital IT solutions, robotics, IT and consulting companies.

What we heard from the participants was that several are still struggling with how to store and analyze data and still have yet to focus on how to actually get the data there. That said, several data integration challenges emerged in discussions.

As the session evolved, an interesting use case surfaced describing a hosted environment that deals with, on the front end, collecting millions of OLTP transactions from millions of customers, including large supermarket chains, and on the back end deals with the need to provide real-time business insights from the data, analysis of the newest collected data, and historic data going back two years and further. This challenge includes both the need to collect the data quickly, but also the ability to deliver to a central repository that will help with data analysis. It is cases like this where a fast, robust data integration solution can provided added-value by enabling the data delivery in near real-time without consuming too many resources on the front end.

Overall, the session flushed out the problems that organizations are facing today when dealing with large and continuous amounts of Big Data. It was clear that these organizations have a long way to go and face many challenges along their journey to address their needs. It was satisfying to help them identify best practices and approaches for managing efficient data delivery and analytics.

Ola Mayer

@attunity

No comments: