Principle and realization of tensorflow intelligent robot

This article is from the author Li Jiaxuan on the gitchat on the wonderful sharing, "read the original" to see what you and the author Exchange issues

The first part explains the principle of natural language processing, the second part explains the realization principle, solution and challenge of the chatting robot, and finally explains the model structure with seq2seq+attention mechanism. The third section explains how to train a chat robot starting from 0. first, the principle of natural language processing

Artificial intelligence to understand the principle of natural language, to answer this question, the first need to define the meaning of the problem, one is a narrow sense of how to use computers to process and analyze natural language, the other is to understand the broad sense of "artificial intelligence", "natural language" and "understanding" meaning.

Let's start with a broad perspective. Get a clear idea of these nouns.

Natural language: The language of communication and communication that has been invented and evolved in human society. Artificial intelligence in the understanding of things (not just natural language), often contains two levels: first, research content, and second, methodology.

The research contents are mainly the current research topics, such as knowledge map, CV, speech recognition, NLP and so on.

Methodology refers to the realization of artificial intelligence methods, there are three main types: symbolism, connectedness, and behavioral doctrine.

Symbolism is the use of mathematical logic reasoning to simulate human thinking intelligence, such as the implementation of expert systems. The most common neural network model is the bionic study of the human brain by the coupling doctrine.

The behavioral doctrine focuses on the predictable human behavior, and believes that human beings get self-adaptability by interacting with the outside environment, and the algorithms involved are genetic algorithm, intensive learning and so on.

The existing NLP is mainly dealt with by the combination of rules and statistics. Its rules are biased towards the perspective of symbolism;

While the statistical side is inclined to dig general law, belongs to induction, the current method, such as the natural language using Word vector method characterization, and then access to the neural network training, that is, the idea of coupling doctrine.

Understanding: There is always a debate about whether a machine can really understand language. Put aside this question, let's see what human understanding of language is. In fact, humans do not necessarily do well in understanding this matter.

For example, the north-south side of the "Tofu Brain" cognition is different, two people can talk about the same object will be different understanding.

Therefore, understanding needs to be determined by many factors, such as life experience, common topic, context, conversational environment, knowledge of both sides.

Since for human beings, really understand each other, need so many hidden factors, that for the machine, we'd better not care about whether the machine can really understand the meaning of the problem itself, but as far as possible to let the machine focus on the above factors, to simulate human intelligence.

The narrow level is the main direction of our engineers ' research.

That is to say natural language understanding is a computer to process and analyze natural language, it involves linguistics (word, part of speech, grammar) and computer science (model/algorithm) category.

From the linguistic point of view, the research direction includes stemming, speech reduction, Word segmentation, POS tagging, named entity recognition, speech disambiguation, syntactic analysis, discourse analysis and so on.

This is the basic category of research, on the basis of the research content, oriented to the specific text processing applications, such as machine translation, Text Digest, emotion classification, question and answer system, chat robot and so on.

In the research of computer algorithms, it is generally based on the combination of rules and statistics, that is, the combination of rationalism and experiential doctrine.

Natural language is essentially a symbolic system, so there are certain rules to be found, but its complexity determines that no rules can conflict with each other and cover all linguistic phenomena.

After the consummation of the large-scale corpus and the popularity of the statistical machine learning method, the burden of many manual rules is omitted, so that the model generates the characteristic automatically.

Therefore, the NLP we are studying is the use of mathematical and machine learning methods to model language. It can be said that NLP is not to achieve true natural language understanding, but to consider language as a computational task. Second, the realization principle of the chat robot, the solution and the challenge

We start with the classification and implementation principles of the chat robot respectively.

At present, the chat robot can be divided into a retrieval-based model (retrieval-based Models) and a generative model (generative Models) according to the mode of dialogue.

The retrieval-based model has a predefined set of answers, and we need to design heuristic rules that can pick out the right answers based on the input questions and context.

The generated model does not rely on a predefined set of answers, but instead generates a new answer based on the input questions and context.

Chat robot of these two technical routes, from the long-term perspective of the current technology are still at the bottom of the mountain, the two technical routes of similarities and differences and advantages are as follows:

The benefits of a retrieval-based model:

The sentence is good readability

A strong diversity of answer sentences

There is an irrelevant answer, easy to analyze and locate the bug

But it's a disadvantage: you need to sort out the candidate's results, make a choice.

Based on the benefits of the built-in model:

End-to-end training, relatively easy to implement

Avoid maintaining a large q-a data set

Eliminates the need for additional tuning of each module to avoid the error cascade effect between each module

But its disadvantage is that it is difficult to guarantee that the resulting results are readable and varied.

Therefore, the common challenges faced by these methods are:

How to use the information from previous rounds of conversation to apply to the conversation in the round

Merge the contents of the existing knowledge base in

Can be personalized, thousand people thousand face.

This is a bit like our information retrieval system, which wants to do better in the vertical domain, and wants to have different sort preferences for different people's query.

From the angle of application purposes, can be divided into target-driven (Goal driven), applied to the customer service assistant, etc., in the closed topic scene; no target driver (non-goal driven), applied in the open topic scenario, this is a subject that can be discussed is not limited to, But the need for a robot has a certain basic knowledge.

Although most of the industrial applications are based on the retrieval model, belonging to the target-driven, such as: Ali Xiaomi, the application of the topic area is narrower, slightly expand the topic, it will be irrelevant reply or irrelevant. The following figure:

A generative model in an open topic scenario should be the most intelligent and consistent with our expected chat bots. So in conclusion:

Target of intelligent chat robot:

and human beings can continue to communicate.

Can give a proper answer to different questions.

Given the diversity of human individuality, the answer to the opposite sex (for example, the same question, the answer to different groups of men, women and young people should vary slightly)

So for an intelligent robot, its chat function in which should be in what position. First of all, the chat should be a basic module, second, the chat should be done with the completion of the task of the module has a good collaboration; Finally, the chat should make the robot look like your friend, not your agent or assistant.

From the above point of view, there are some often confused with the chat robot concept, but also some chat system peripheral products:

QA question and Answer system: A field robot that answers factual questions (such as how high The Everest is) and non-factual questions (such as why, how, opinion, and so on).

Dialog System dialog: This is mostly target-driven, but in recent years it has been slowly accepting the chat robot function

Online customer Service: For example, Taobao Xiaomi, which in most cases resembles an automatic FAQ.

Therefore, although the chat system is aimed at the general direction of the text comprehension, but the target difference determines the technical route will be biased, but the chat function is a basic function.

Intelligent chat robots can be used to learn from the data sets in the field of peripheral system research:

Questions and answers on non-factual issues

A community-based question and answer system (such as Baidu knows that there is a strong match between questions and answers, and a question with multiple answers with ratings, sorting)

Mining some good QA corpus from the online system

So how to evaluate a chat robot is good or bad. The most important is the relevance of the question and answer sentence, that is, the essence is: short-text correlation calculation. But be aware that:

Similarities and correlations are different. The various methods used for similarity calculations do not apply to correlations. We need to establish a short-text correlation calculation method.

Correlation calculation has some methods that continue in the construction of early chat bots:

Statistics on the total occurrence of words

Correlation calculation based on machine translation

Similarity calculation of subject model (LDA)

The deep learning methods currently used on chat bots are as follows:

Word2vec, Glove

CNN, LSTM, GRU

Seq2seq

Attention mechanism