Data Crawler analysis of big data related posts in pull-hook net

Analysis of recruitment data related to big data of pull hook net

Audience: Job data for big data-related jobs

Observation Time: 2016.3.28

Data source: Pull Hook Net

1. Purpose of analysis

At present, big data is a very hot topic, by many people's attention and pursuit, the creation of related occupations are also favored by everyone. But what is the big data-related occupation, what kind of requirements, what kind of treatment is not known to the majority, in order to better understand the big data related professional requirements and benefits of this data analysis.

2. Data acquisition

1 data Source: Pull hook net, Hook Net is a professional Internet recruitment platform, focus on the Internet career opportunities, its data is representative;

2 data type: JSON type data;

3 Acquisition method: Python crawler, the input keyword is ' big data ', so the collected data including all big data related job data;

4 Data Volume: 99 pages Total 1476 data.

3. Data preparation

This collection of data a total of 1476, each data has a 50 variable. To focus on the main factors, unnecessary variables are removed during the data preparation process, and due to the large amount of data, a small number of rows with missing values are cleared, and data sets that are easily analyzed and plotted are reconstructed.

4. Data analysis

1. Analysis of the distribution of big data related employment information in various cities nationwide

The distribution of large data recruitment information by cities shows that the recruitment of big data-related occupations in Beijing, Shanghai, Guangzhou, Hangzhou, Shenzhen and other economically more developed regions, especially the capital Beijing, is 3 times times the number of these cities. It is not surprising that the big data is still a new industry, many two or three-line cities of traditional enterprises and companies are still in the wait and see. For the number of Beijing, the individual think mainly related to national policy, entrepreneurial tide, after all, is the capital, can quickly smell to the country to support the development of big data, and the ' Internet + ' entrepreneurial tide has also advanced the company's desire for big data talent.

2. Analysis of major data-related occupational types

The distribution of major data-related job types indicates that big data technology professionals are the most popular, followed by products and operations. Some people say that big data scientists are programmers of the program, from which you can see the ' ability to program ' and the ability to process and mine data, or to occupy important factors. For products and operations, it may be related to the recent discussion of ' portrait portraits ' like ' refinement operations ', and the use of big data-related knowledge to achieve precise marketing. Of course, big data in finance, marketing and other aspects also gradually received attention, personally think this is a trend, after all, big data is only a means, more importantly, how to use big data in various industries, for the industry services.

3. Analysis of skills requirement for big data related job recruitment

It is discussed above that big data tech talent is the most popular, and then go on to see if big data really favors that kind of skill or that language and tool.

By the Bubble distribution chart (the larger the circle, the greater the importance), the top 10 big data tools that are most favored are Hadoop, Java, Spark, Hbase, Hive, Python, Linux, Strom, Shell programming, and MySQL. Both Hadoop and Spark are distributed parallel computing frameworks, which now seem to dominate Hadoop and spark is behind, but Spark has a catch-up trend. Hadoop is implemented by Java, so it's not surprising that Java is behind the queue. HBase is an open-source, distributed, and database, MySQL is an open-source relational database, hive is a data warehouse, Strom is a streaming framework, and Python/shell is a two-script programming language, Linux is an operating system.

If the above figure looks more laborious, let's take a look at the following diagram:

4. Big data related job recruitment to the academic requirements analysis

The major data related to the professional qualifications requirements are mainly undergraduate, followed by college, and for the high degree of Master and doctor seems not too cold. As big data is mainly interested in technical personnel, the practice of the work is relatively high, may be higher education talent instead of the advantage. We can then compare the requirements of the work experience to do in-depth analysis.

5. Big Data related career recruitment to work experience analysis

It is known that big data-related occupations are most favored by people with 3-5 years of work experience, followed by 1-3 and 5-10 years. Compared to the requirements of academic qualifications, it is true that big data-related occupations are favored undergraduates with working experience, rather than the master and doctoral students with high academic qualifications but lack of experience.

6. Salary analysis for big data related occupations

The figure shows that the overall wage level of big data-related occupations still has a large fluctuation, but it is mainly concentrated in the range of 10k-30k. Let's look at the distribution of wages for different job types:

The average wage for big data jobs related to the financial sector is the highest, with a small difference in the average wage between products, technologies and functions, with relatively low wages in the market and in sales and operations, but the average wage is above 10K. Generally speaking, the salary level of big data related occupations fluctuates with the work experience, but the salary is relatively high.

7. Analysis of welfare benefits of big data related occupations

From the benefits of a company with big data talent recruitment needs, the most of which is the basic protection of five risk one gold, followed by paid leave, flexible work, double Hugh, year-end award, performance award, and so on, overall these companies are good, but from these data can be seen, compared to some large state-owned enterprises, These companies are more concerned about the settlement of accounts and other students, not mentioned.

8. Analysis of corporate financing situation with big data recruitment needs

From the know, there are big data recruitment needs of companies listed companies still occupy the largest proportion, ranked in front of the several are also growth-type or mature-funded companies, ranked in the back of a few companies in addition to a mature D-round, the others have no financing, or do not need financing, It can be explained that the desire for big data talent is higher for listed companies or for growth companies that have just got financing.

5. Conclusion

From the analysis of the surface, we can draw a few important conclusions:

A. Big data is just north-canton and other economically developed cities developed very hot new industries, two or three-line city has yet to be developed, so to find big data related work to go to the North Canton Bar;

B. Big data-related jobs mainly in technology, products, operations-oriented, and technology occupies most of the country, and technical skills required mainly to Hadoop/java/spark/hbase/hive/python/mysql/strom/shell, etc. Therefore, to engage in big data-related posts from learning these skills start;

C. Big data related occupations to the requirements of academic qualifications are mainly undergraduate, even if the college education is also very popular, and the Doctor and master is not favored, this is a requirement of work experience for the industry, so even if you are not high, want to engage in big data related work is not a problem;

D. Big Data related jobs salary is still relatively high, welfare benefits are also good, including financial big data talent wages the highest;

Currently recruiting big data talent companies are mainly listed companies and growth-oriented financing companies.

6, have the problem exchange can pay attention to dataanswer Big data http://www.dataanswer.top

Data Crawler analysis of big data related posts in pull-hook net