Is the data thinking important in the era of big data, or is it important to the relevant technology?

Technology to a certain extent, and gradually find their own bottlenecks. Can't help but start to think about this aspect of the problem! In the big Data age, is the corresponding data analysis technology important, or the corresponding data thinking important?

Let's start with data thinking! What is big data thinking, personal feeling should be a kind of internet thinking. is considered to be comprehensive, rather than partial. is to take into account multidimensional, rather than a single dimension. Instead of making decisions on the forehead, let the data speak and use the data to make decisions.

Say the 1th first, consider the whole, not the partial. It is well known that the mobile Internet has spawned big data generation. Every single day the sum of the data that can be done by mobile phone is a huge amount. With these unstructured data, the first thing we are dealing with is how to deal with this data, which involves the storage of data, the problem of reading. Because of the unstructured data. Traditional processing technology will not be able to play a very good role. We can predict the behavior of a large number of users by not counting the data of several users in a single sample. The global data is needed here. First, this is the 1th difference between big data versus other technologies.

For the 2nd, consider multidimensional, not a single dimension. As everyone can see, the ads are now starting to be recommended two times based on our browsing duration. In the multidimensional analysis, will no longer only focus on the user's browsing data, and the corresponding will have the user's text information, friends comments, have purchased product information, with Kapin times .... When the user data for multiple dimensions of the integration analysis, you can achieve precision marketing. Thus breaking the traditional one-dimensional marketing of passive propaganda does not applaud the characteristics.

Finally, look at the 3rd, let the data speak, use the data to make decisions. There is a need to mention a software R, the traditional industry's business statistics are mostly done by it, but his data presents a bit of limitations. When we present some data in the form of a chart, we can make a new year's annual plan based on that data. Without making a decision on the forehead. And now the application of data visualization is also very much, the actual application also shows the corresponding value.

Say the above three points, and then say big data technology.

The first thing to mention is Hadoop, a distributed storage that is now in use in most enterprises, while its distributed storage shortens the user's read time. Next-generation technology, spark, is the equivalent of storing from a Hadoop hard drive and moving to memory storage. It is well known that memory reads much faster than hard drives.

The next thing to say is SAS and R, both of which have their own advantages, SAS as a professional data statistics software, can be said in the large data volume when processing has significant advantages. But in the case of more than 1T of data, its advantages are no longer so obvious. While the R software has a great advantage in drawing, it has a very important position in the visualization of data. But helpless, r software in statistics slightly inferior to SAS one chip. In the same software as SAS, SPSS personally feel that the amount of data in the hour will be a great use.

There is data acquisition, Rcurl and the corresponding crawler technology, and in the big data period, the site in the anti-crawler will certainly do the measures to protect their existing data.

There is ETL, this point, personal feeling will be in the big data fall in the very important position. Because most of the domestic data need to be cleaned, as my mentor said in a word, big data in China, the biggest difficulty is how to identify false data. Use real data to analyze the results we are going to get.

Well, so far, the above is the knowledge of the big data that you have learned, respectively, in the thinking and technical aspects of understanding. But the problem is back, for the enterprise, profit is the most basic choice, in the end is to choose a big data thinking people or a lot of big data technology people? For themselves, the next step is to continue to learn the technology, or grasp the corresponding thinking.

Thinking or technology, or left or right? Or both need fusion, but the topic is back, the work is very realistic problem, pre-sales, consulting, research and development, architecture, implementation, engineering .... In the end how to choose, or do not choose, first calm down to learn, until the job of the study after graduation to choose work.

In the face of career change, from the communication training industry to the Internet industry Big Data Transformation, although I do not know the way ahead, still some confused, but since the choice ahead, then only trials and hardships, refueling!

This article is from the "Data Mining and Visualization" blog, reproduced please contact the author!

Is the data thinking important in the era of big data, or is it important to the relevant technology?