What skills/Algorithmic engineers are required for machine learning

https://zhuanlan.zhihu.com/p/21276788

Objective

Originally this title I think is the skill of algorithmic engineer, but I think if add machine learning in the title, the estimated point of people will be more, so the title into this, hehe, and is indexed by the search engine when more a popular word, estimated exposure will be more points. But rest assured, the article is not tricky, we are serious. Today talk about machine learning this last two years the computer field of the hottest topic, this is not a machine learning technology article, just tell everyone machine learning inside the pit is too much, and many have not started or just getting started friends, in fact, in front of you is a big pit, if you are inspirational to go down on this road, Please be prepared mentally.

We learn the purpose of machine learning

To tell the truth, the majority of people in various classes to learn machine learning, learning big data, in the final analysis or hope to find a good job, get a higher salary, of course, there is a part of the reason is that they are more interested in this aspect, want to know more deeply about this field. I personally think that the first reason is more important.

What are we talking about when we talk about machine learning?

First, let's see what a machine learning system looks like.

Almost all machine learning systems are made up of the above-mentioned system diagrams, different supervised system training data may require manual intervention rather than supervised systems do not require human intervention, simply to give a batch of training data to this machine learning model to learn , to obtain a predictive model , and then use this predictive model to predict new unknown data. Now online machine learning articles, blogs are everywhere, the market all kinds of books are everywhere, and the current education is the most fire field is this, a variety of machine learning online education classes, tuition is very expensive. But you find no, all these talk about machine learning is talking about models, what "in-depth understanding of XXX Model", "may be the best understanding of XXX article", "machine learning is not difficult, XXXX model detailed" and other articles and books everywhere. Various introduction to logistic regression, deep learning, neural network, SVM support vector machine, BP neural network, convolutional neural network ..... Wait, wait. So, when we talk about machine learning, we're actually talking about machine learning models, which are all kinds of machine learning algorithms. And everyone thinks that as long as the theory of model and algorithm is learned, it is the expert of machine learning. I believe most people think so.

Xiaoming became a machine learning "expert"

There is a child, is engaged in computer, called Xiao Ming, saw Alphago abuse Li Shishi video, although he did not understand go, but he was shocked, determined to learn this legendary machine learning. So everywhere in the online search tutorials, find blog articles, find books, good study for half a year, finally feel that they started. Each machine learning model algorithm can tell a why.

Do not know how many people there at this stage?

But Xiao Ming also want to go further, so began to study various models of code and tools, Hadoop and spark that is standard, but also a variety of search articles, all kinds of books, all kinds of online classes, fortunately these things a lot of a lot of, especially now online classes, if there is no big data processing class, If you don't have a Hadoop class, don't open it. All the way down, the big six months later, and finally Xiao Ming felt himself learned, theory also has, big data processing tools will also, simply invincible!

And how many people are at this stage? And thought he'd already learned the machine. At this stage, if you learn well, then you can have a class to teach others machine learning. But if you think so you can find a company to do algorithmic engineers, then tell you, the pattern Tucson broken, righteousness five!

Xiao Ming because of a strong theoretical knowledge, can deduce all the formula, and will Hadoop,spark, coupled with their ability to express, very easy seconds a few interviewers into a large company, is in an e-commerce search algorithm engineer, the monthly salary is high, finally can a show fist, the boss gave him a task, Use your awesome knowledge to increase your search rate by 1%. If you were xiaoming, what would you do if you had just learned a lesson from a machine? Are you stupid?

Machine learning is more than a model

The reason for this problem is that all people think that machine learning model is machine learning itself, think of those algorithms understand that is machine learning Daniel, but in fact it is not the case at all. Who is playing the model? The model is invented by scientists , is a large company of various scientists , researchers invented, the invention will be out of the paper, they are used to abuse our IQ, under normal circumstances, you can not invent the model bar , can not look down, you can go to the academic road)? You can't change the model, can you? So, learn the model, just just getting started, not even the introduction of the company's so many algorithmic engineers are doing? Let's take a search-sequencing algorithm engineer, for example, what are they doing?

Observe data---> Find features---> Design algorithms---> Algorithm validation---> Wash data---> Engineering---> On-line viewing results--->goto observation data

and a mature system, the general model has probably been determined, if the effect is not particularly bad will not change the model, such as a company's search sorting system with machine learning logistic regression model, you have to change to other models is generally not possible, then only to do some of the characteristics of the supplement. Well, let's take a look at what a machine-learning algorithm engineer can do with this process.

Observation data

Xiao Ming every day in the station to see data, check data, look at the table, draw curves, found like sales , collection , click and so on this can think of features have been used, so consumed for three months, no progress, people have collapsed, come so long, Machine learning code hair didn't see it. Fourth month, he found a little problem, he found some goods, comments and so on are very good, feel the product quality is also good, but is not on the sales, so the old platoon behind, so he put these comments are five-star, but the sales of poor products filtered out, want to see what they have in common. What do you say you want to be able to observe the data phase? Hehe, can only tell you, need data sensitivity, actually is to tell you need comprehensive ability, need experience, need the ability of product manager. In addition to these, you also need to be able to readily compile script code, encounter some data need preliminary processing, may need to readily code processing, and make up fast, because the code may be used one or two times, so need more powerful scripting language ability, then python at least familiar with it, the shell will be.

Find Features

Data observation down to find the problem, now to look for features, to find features, that is, what factors lead to sales do not go, first of all, the need for imagination, and then to verify your imagination. Xiao Ming's imagination, even so, also engaged for one months only to find that these products have a common feature, that is, the pictures are rotten, so that people do not want to see a point. If you can add the image quality to the sorting factor, is it a magical thing? Image quality as a feature, this has not been done before, and finally found a feature. So at this stage, after all, everyone's imagination is limited, more experience, to find the characteristics that match the current scene.

Design Algorithms

The feature is found, but how do you add this feature to the sort model? Picture good, how good, how do these machines understand? If you can't turn the image quality into a mathematical vector, you'll never be able to add it to the sort model. This phase is the real test of algorithmic engineers, that is to quantify the characteristics, Xiao Ming observed that the more beautiful images tend to change more color, and poor quality of the picture is often the color changes , so he thought of a way, First, the image data Fourier transform into a frequency domain data, according to the nature of the Fourier transform, high-frequency part of the amplitude is very obvious, if the low-frequency part is high, the color change is not obvious , this and the observed image information basically can match, such an image of the good or bad, You can use the Fourier transform after the high-frequency portion of the expression, and then do some normalization of the changes, the image is quantified, vectorization can be added to the sequencing model. This step, you may use your learning machine learning model, but certainly only a small part, most of the time you need to build a mathematical model based on the current scene, rather than the machine learning model, you say this phase requires what skills? Although the examples I cite here are extreme, mathematical abstraction, mathematical modelling and the skillful use of mathematical tools are essential, and the same need for strong programming skills, which is not a previous scripting ability, is real deal's computer algorithm programming ability.

Algorithm validation

Algorithm is designed, but also to design an algorithm off-line verification method to prove to your boss to see that my algorithm is effective, or so many opportunities to let you go to the line to try Ah, this step is a combination of various comprehensive capabilities, the key is in this step, you have to use a popular language in theory to persuade your boss , what kind of ability is this? Strong ability of language expression. In addition to this you need to design an on-line AB test plan, can be very good to test whether your algorithm is really effective.

Wash data

Features found, the algorithm is also designed to almost reflect the characteristics of physical activity, that is to wash the data, this is a compulsory course for the algorithm engineer, the data is not what you want to look like what he looks like, so to make the data you want to look like, and then remove the invalid data is a physical activity. Like the above example, the first may be the size of the picture is not the same, to become a size before the transformation, some products have multiple pictures, you may need to find the best quality of re-processing and so on. This stage is also to scripting language processing ability , but also need to master some of the use of data processing tools, the key to have enough patience and confidence, of course, the essential is the excellent programming ability.

Engineered

All right, you've crossed all the holes in the front. To this step, hehe, algorithm design is finished, the data is ready, estimated half a year passed, then quickly put on the line up, you think with a bunch of script can go online ah, have to consider engineering, if your algorithm embedded in the original system, if you ensure that the efficiency of your algorithm, Do not run a day, the robustness of the code should also consider Ah, if it is an online algorithm, but also to consider performance, do not put the memory dry. This step, you really use the machine you learn the Hadoop,spark tool, read the above said, to complete the project of this step, have what ability without I say, this is a standard software development engineer necessary skills, or senior development engineer OH.

On-line viewing effect

All done, back and forth 10 months, finally can go online, well, the real test came, look at the effect of the line, product manager said, do a AB test bar, the results hehe, the CTR reduced, Xiao Ming Ah! This 10 months of busy down the click rate has also declined??? The boss is not going to scold you for dying, so you have to have a strong anti-shock ability. Oh, quickly off the line, from the beginning to see where the problem, and spent one months to modify the algorithm, re-online, ah, this is good, click-through increase of 0.2%, continue to work hard, see if there is nothing to dig, so, you goto the data to see the step. Don't look at this 0.2, the big data collection, the increase of 0.2 is already very good to improve, so spend so much money, foster algorithm engineer, if a year can make a few 0.2, that is the truth.

Let's summarize

Above so many processes, rely on a person complete really a bit difficult, I said a bit exaggerated, some of the steps in the middle is someone to cooperate with the data when there is a product manager to cooperate with you, when the data is washed with the data engineer to cooperate with you, Engineering when the system engineers with you , But as an algorithmic engineer for machine learning, you have to hold on to the whole process, so even if you're a person you should be able to complete the process. This is just a standard algorithm engineer should have the ability, of course, I here is the search algorithm for example, the other algorithm engineers are not too much, always run but the above several processes, of course, if you are a cow, can modify the machine according to the scene model of learning, and even can think of a model, it is more powerful. OK, let's take a look at the key points above, and let's see what kind of skills an algorithmic engineer needs.

• Data sensitivity, observation
• Ability of mathematical abstraction, mathematical modeling and proficiency in mathematical tools
• Ability to readily compile script code, strong computer algorithm programming ability, the quality of advanced development engineers
• Imagination, patience and confidence, strong verbal ability, anti-attack ability

Then, there is a very important point, you need to be very smart, of course, if you can do the above points, basically will be very smart, if you can do so, but those machine learning models, theories and tools are not so important, because those are just knowledge and tools, can learn at any time. You said, these are by looking at several blogs, read a few books, the last few lessons can have?? Of course, we're talking about the general situation here, and if you're focused on doing research, then you need to increase the proficiency level of the above skills. Learn machine learning, self-motivation to do algorithmic engineer you, ready to step on these pits??

Welcome to my public number, mainly talk about search, recommendation, advertising technology, and nonsense. The article will be sent first here:) scan or search for XJJ267 or search for the language of West Gaga .

What skills/Algorithmic engineers are required for machine learning