The Path of Big Data: Cooperation, Innovation, Win-Win Partnerships

Statistics show that the amount of internet data doubles every two years, and more than 90% of current data has been produced in the last few years. Massive scale is only one aspect of big data; other aspects are speed, variety, and value. The last of these is key to the future of big data.

Necessary Conditions for Big-Data Development

Big data can be developed in terms of data sourcing, data transactions, and creating value out of data. Recent years have seen the rise of social networks, the development of M2M, and the popularity of mobile internet. As a result, many valuable data sources have emerged, and these have laid the foundation for big-data development. There are two important symbols of the advent of the big-data era: 1) the emergence of a large number of specialized data traders, and 2) the emergence of an industry related to data trading that includes data collection, clearing-up, analysis, and application. Central to big-data development is enabling users to create new value from vast amounts of unstructured and semi-structured data. The value of data is the driving force behind data transactions.

IBM, Oracle, and SAP have invested heavily in acquiring data management and analysis companies in recent years. Thanks to the efforts of these internet giants, data-analysis technology has matured. In June 2013, Edward Snowden exposed the U.S. “Prism” plan, which shows how mature big-data technology is being used in the service of national security. However, people are yet to experience the benefits of big-data technology in their everyday lives because there is little value in the data transactions.

The development of big-data technology promotes the deployment of cloud computing which, in turn, increases market expectations about the value created by data. The market has finally seen the profit potential of big data and cloud computing, and the market for cloud computing seems to have exploded overnight. System integrators are working with local governments to build cloud data centers and smart cities. Industry leaders are establishing hybrid cloud standards for their own industries in order to build up industrial cloud platforms. Public clouds have also come into being. IT giants are trying their best to obtain a public cloud license in China. Because of market expectations of the value of big data, cloud computing has finally been implemented, five years after it first emerged. The concept of smart city has been widely accepted, and cloud computing infrastructure has been prepared as the hardware foundation for big-data applications. Because of the expectations for ROI in cloud computing, big data must produce great value from its applications. Now, the emphasis is on how data can create such value. 

Data Integration and Openness

According to a survey of 800 business and IT managers, conducted by Connotate in 2012, 60% of respondents said that it is too early yet to be certain about the ROI in big data. At present, big data is not open enough; that is, big data is in the hands of different sectors and enterprises that are not ready to share data. The inherent laws of big data have to be determined through study of data correlation, and this relies on data authenticity and universality. Data sharing is a major area of concern in big-data development.

Obama’s success in the 2012 presidential campaign benefited from data integration. The campaign team employed a data-mining group who helped raise $1 billion through mass data mining. Campaign advertising was also made 14% more efficient through data mining. The data-mining group provided the raw data used to create detailed models of voters in swing states and calculated Obama’s chances of success in these states by simulating the elections 66,000 times every night. This means the campaign could channel resources efficiently. Data integration differentiated the Obama campaign from the Romney one. The data-mining group of the Obama campaign also realized that data was scattered across too many different databases, so in the first 18 months, they created a single data system that could integrate all the information coming from pollsters, donors, on-site staff, consumer databases, and social media as well as information about the main Democrat voters in swing states. This huge database helped the Obama campaign pinpoint potential voters and capture their attention. It also helped the campaign team predict what types of people could be persuaded by specific stories. Obama’s campaign manager, Jim Messina, said that there had been very few hypotheses that were put forward without data support.

In March 2012, the Obama administration announced that it would invest $200 million into big-data research and development. Big data has been defined as a national strategy; mass data and its related applications will become an important part of a country’s overall strength. Data sharing is also one of the goals of smart cities in China.

Business Model for Big Data

As cloud computing, big data, and related businesses mature, more software developers will use cross-industry big-data platforms to build applications that create value from this data. This is becoming easier. Data owners are more willing to make this happen so that they can generate extra income for minimal cost. Big-data equipment vendors are also willing to see this happen because they need applications that attract consumers to buy their devices. Vendors generate more profit by entering into in-depth win-win partnerships than by merely focusing on selling devices. Some far-sighted vendors have started supporting software developers by providing funds, technical support, and stock. There is also a growing need for data-analysis applications in the enterprise market, and innovative developers of enterprise data applications will have a central role in the entire big-data industry chain.

In the forthcoming era of big data, enterprises that have mass data, strong data analysis capability, and innovative software developers will dominate the big-data industry chain. Social networks, mobile internet, informatization enterprises, and telecom operators are all creators of mass data. Facebook boasts a user base of 850 million; Taobao has 370 million registered users; and Tencent has 400 million WeChat users. The data produced by these huge user bases may be a source of enormous business opportunities someday. In the near future, owners of mass data will either grow to be data-analysis providers or cooperate with enterprises such as IBM and ZTE in upstream/downstream business relationships. The big-data industry chain will grow surprisingly fast after a certain flashpoint.

Potential Drawbacks of Big Data

Instead of data being randomly sampled, all data can be collected and retained so that analysis of this data is more accurate and inferences do not need to be made for the whole data set. This decision-making model is more accurate because it cuts out interference that might come with personal emotions, psychological motivations, and sampling errors. The accuracy of big data depends on the methods used to collect it. Any problem in the method of collection can lead to flaws in the data. As a consequence, decisions based on this data may also be flawed.

Just as too little information may be useless, too much data whose accuracy and value cannot be determined is also useless and may even be detrimental in scenarios where it is the basis for important judgments. The big-data theory is based on the fact that all data is true. What if a data provider forges data? This can be very harmful because the prejudice and filtering process of data providers are not controllable. Wall Street investment banks and top European and American rating agencies were the first to accept the concept of big data and owned perfect databases; however, they have suffered at times from critical errors of judgment based on erroneous big data. This reveals the limitations of big data. It is necessary to be aware of the potential drawbacks of big data while continuing to develop it. 

The economic value of big data has been recognized, and big-data technology is maturing. Big data will reach a critical turning point when data has been integrated and regulated.