Data Deluge and Its Analysis Issues

Abstract

Current availability of enormous amount of data is mainly due to technological advances. They are useful drawing inferences for creating new businesses, formulation of new policies or revising existing ones, etc. However, much of analyses are performed either by subject domain experts implementing mathematical and computational models incorrectly or by mathematical and computational professionals, purely on data driven basis without paying required attention to the subject domain knowledge. Both of these exercises often result in incorrect inferences and therefore they may harm the society, especially when their inferences are used in practice. We argue that, in order to get valid inferences these two parties should work together. Here we briefly discuss some of the issues that the large-scale data analyses should take into account, especially in open data and big data. We also briefly discuss our solutions that are rather simple to implement.