Data Fusion: Dealing with the scale of distributed computing

We are living in a data explosion. More information is being created now than ever before. There are more networked devices than ever before. This trend is likely to continue into the future. While this makes it easy for companies to collect data, it also brings challenges of sheer scale. How do companies deal with data from millions, possibly billions of sources?

In order to gain insight into the frontiers of distributed data collection, JeffFrick (@JeffFrick), the co-host of the Silicon Steel Media Team, visited the Chief Data Scientist event in San Francisco, California. There, he met Sam Lightstone, an outstanding engineer and chief architect of Data Warehouse at IBM.

The discussion started with a recently announced concept technology called “Data Convergence”. Lightstone explained that data convergence is a brand new idea they have brewed at IBM. It comes from the consciousness that a large amount of data will come from distributed sources such as mobile phones, cars, smart glasses, etc.

“This is really a lot of data,” Lightstone said.

The idea behind data fusion is to keep the data in place. Wrightstone described it as allowing data sources to find each other and collaborate on data science problems in a computing grid.

Lightstone mentioned a huge advantage of this concept, it can bring hundreds of thousands, even millions of processors to process the data it lives in. He called this a very powerful and necessary concept. Such a network must be automatic in order to expand to hundreds of thousands of devices.

The complexity of such a system is too much for humans. Lightstone said his goal is to make this kind of automatic and flexible, state-adaptable device connect to it. He said that with the convergence of data, they hope to leverage data science for IoT, enterprise and cloud use cases.

* Disclosure: Corinium Global Intelligence and other companies sponsored some chief data scientists, and the US part of the CUBE on silicon steel media. Neither CoriniumGlobal Intelligence nor other sponsors have editorial control.

Solemnly declare: The copyright of this article belongs to the original author. The reprinted article is only for the purpose of disseminating more information. If the author’s information is incorrectly marked, please contact us for modification or deletion as soon as possible. Thank you.


Related Posts