Teaching machines to understand us let the machine understand our belief in three natural language learning and deep learning

Language Learning

Natural language Learning

Facebook's New York office is a three-minute stroll up Broadway from LeCun's office at NYU, on both floors of a building co Nstructed as a department store in the early 20th century. Workers is packed more densely into the open plan than they is at Facebook's headquarters in Menlo Park, California, but They can still is seen gliding on articulated skateboards past notices for weekly beer pong. Almost half of LeCun ' s team of leading AI researchers works here, with the rest at Facebook's California campus or an Offi CE in Paris. Many of them is trying to make neural networks better at understanding language. "I ' ve hired all the people working in this I could," says LeCun.

A three-minute walk along Broadway from LeCun's office in New York University, the New York office of Facebook, built in early 20th century, is a department store, and the Office is on the second floor of the building. The staff huddled together in the open layout, more crowded than Facebook's headquarters in Menlo Park, California, but could still see signs that they had slipped on their skates to the weekly beer party. Almost half of the main AI researchers at the LeCun team work here, leaving the rest of the office in the California campus of Facebook or in Paris. Many of them are trying to make neural networks better able to understand natural language. "I've hired everyone who has a job in the field," LeCun said. ”

A Neural network can "learn" words by spooling through text and calculating what each word it encounters could has been PR Edicted from the words before or after it. By doing this, the software learns to represent every word as a vector that indicates it relationship to other WORDS-A PR Ocess that uncannily captures concepts in language. The difference between the vectors for "king" and "Queen" are the same as for "husband" and "wife," for example. The vectors for "paper" and "cardboard" is close together, and those for "large" and "big" is even closer.

The way the neural network "learns" the language is to scan these words and calculate how each word encountered is predicted by the preceding or subsequent words. In this way, the software represents each word as a vector, representing the relationship with other words, which is the mysterious process of capturing concepts in a language. For example, the difference between "King" and "queen" vectors is the same as the difference between "husband" and "wife". "Paper" and "cardboard" vectors should be very similar, "large" and "big" vector should also be the same.

The same approach works for whole sentences (Hinton says it generates "thought vectors"), and Google are looking at using I T to bolster its automatic translation service. A recent paper from researchers at a Chinese university and Microsoft ' s Beijing Lab used a version of the vector technique To make software this beats some humans on iq-test questions requiring an understanding of synonyms, antonyms, and analog ies.

For the whole sentence, there is the same approach (Hinton says it produces "thought vectors"), and Google wants to use it to support automated translation services. A recent article in a Chinese university and a Microsoft Beijing Research Institute has used this vector technology to create software that defeats a number of human participants in an IQ test that requires understanding synonyms, antonyms, and analogies.

LeCun ' s group is working on going further. "Language in itself are not so complicated," he says. "What's complicated is have a deep understanding of language and the world that gives you common sense. That's what we ' re really interested in building into machines. " LeCun means common sense as Aristotle used the term:the ability to understand basic physical reality. He wants a computer to grasp, the sentence "Yann picked up the bottle and walked out of the" the "means the bottle lef T with him. Facebook's researchers has invented a deep-learning system called a memory network that displays what's May is the early St Irrings of common sense.

The work plan of the LeCun group is more long-term. He said: "The language itself is not so complicated, it is complex to have a deep understanding of the language and the whole world, which will give you common sense." This is what we are really interested in and can be integrated into the machine. "LeCun's common sense is like the meaning of the term Aristotle refers to: the ability to understand basic physical reality." He wanted a computer to understand the sentence "Yann picked up the bottle and walked out of the room," knowing that the bottle followed him out of the room. Facebook researchers have created a deep learning system called the Memory Network, which may be the early beginnings of common sense.

A memory network is a neural network with a memory bank bolted in to store facts it had learned so they don ' t get washed a The every time it takes in fresh data. The Facebook AI Lab has created versions that can answer simple common-sense questions about text they had never seen BEF Ore. For example, when researchers gave a memory network a very simplified summary of the plot of the The Rings, it could a Nswer questions such as "Where is the ring?" and "Where was Frodo before Mount Doom?" It could interpret the simple world described in the text despite have never previously encountered many of the names or objects, such as "Frodo" or "ring."

A memory network is a neural network that comes with a memory inventory, which is used to store the learned facts, so that each time new data comes, it is not washed away. The Facebook AI Lab has developed several versions and has been able to answer some simple common-sense questions that they have never seen. For example, when the researcher gave the memory network a very simplified version of the story of The Lord of the Rings, it could answer like "Where is the ring?" "and" where is Frodo before Mount Doom? " It can explain the simple world described in the text, although it has never before encountered these names and objects, such as "Frodo" or "ring".

The software learned its rudimentary common sense by being shown what to answer questions about a simple text in which C Haracters do things in a series of rooms, such as "Fred moved to the bedroom and Joe went to the kitchen." But LeCun wants to expose the software to texts that is far better at capturing the complexity of life and the things a V Irtual Assistant might need to do. A Virtual Concierge called Money-penny that Facebook was expected to release could being one source of that data. The assistant is said to being powered by a team of human operators who'll help people does things like make restaurant Reser Vations. LeCun ' s team could has a memory network watch over Moneypenny's shoulder before eventually letting it learn by interactin G with humans for itself.

How does the software learn these basic common sense? Before giving an example of how to answer a simple question, give a simple text with a character in a series of space to do things, such as "Frodo moved to the bedroom, Joe to the kitchen." But LeCun wants the software to receive more complex text than to describe the complexities of life, or to receive what a virtual assistant needs to do. A virtual gatekeeper, called Money-penny, that Facebook wants to launch, can be a source of this data. This assistant is maintained by a team that can help people do things like ordering meals. The LeCun team can have a memory network to receive Money-penny's work, of course, before allowing it to interact with humans before learning.

Building something that can hold even a basic, narrowly focused conversation still requires significant work. For example, neural networks has shown only very simple reasoning, and researchers haven ' t figured off how they might is Taught to make plans, says LeCun. But results from the work that have been done with the technology so far leave him confident about where things is going. "The revolution is on the the," he says.

It also requires a lot of work to make algorithms that are capable of making basic conversations with little conversation. For example, LeCun says, neural networks have only a simple reasoning ability, and researchers have yet to figure out how to teach the Web to plan. But the results of the work that has been done have given him confidence in the progress of the matter, he said: "The revolution is on the way." ”

Some people is less sure. Deep-learning software so far have displayed only the simplest capabilities required for what we would recognize as Convers ation, says Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence in Seattle. The logic and planning capabilities still needed, he says, is very diferent from the things neural networks has been DOI ng best:digesting sequences of pixels or acoustic waveforms to decide which image category or word they represent. "The problems of understanding natural language is not reducible in the same," he says.

Some people are not so sure. Oren Etzioni, CEO of the Seattle Allen Ai Institute, says that the ability to talk in depth learning software is just the most basic and simple, and that the logic and planning that is still needed is very different from what a neural network can do: receiving a pixel sequence or a voice waveform to determine which category the image belongs to , which word the voice represents. "The question of understanding natural language cannot be simplified in the same way," he said. ”

Gary Marcus, a professor of psychology and neural in NYU who have studied how humans learn language and recently St Arted an artificial-intelligence company called Geometric Intelligence, thinks LeCun underestimates what hard it would be f or existing software to pick up language and common sense. Training the software with large volumes of carefully annotated data are fine for getting it to sort images. But Marcus doubts it can acquire the trickier skills needed for language, where the meanings of words and complex sentence S can flip depending on context. "People would look back on deep learning and say this is a really powerful technique-it's the first time that AI became PRA Ctical, "he says. "They ' ll also say those things required a lot of data, and there were domains where people just never had enough." Marcus thinks language may be one of the those domains. For software to master conversation, it would need to learn more like a toddler who picks it up without explicit instructi On, HE suggests.

Gary Marcus, a professor of psychology and neuroscience at New York University, has studied how humans learn languages, and recently established an AI company called Geometry Intelligence, which he thinks LeCun underestimates the difficulty of learning languages and common sense in existing software. It is possible to classify the image by training the software with a large amount of carefully annotated data. But Marcus doubts that this is not enough for a language that requires more complex techniques, and that the meaning of words and complex sentences can be completely different in a context. "People will look back on deep learning and say it's really powerful technology, which is the first time artificial intelligence has become practical, and people will say that it requires a lot of data, and there are always areas where people will never have enough data," he said. "Marcus that language is such an area. He argues that software that wants to master conversational skills should be more like a toddler who learns without explicit instructions.

Deep belief

Faith in deep learning

At Facebook's headquarters in California, the West Coast members of LeCun ' s team sit close to Mark Zuckerberg and Mike Sch Roepfer, the company's CTO. Facebook's leaders know that LeCun's group is still some the from building something you can talk to, but Schroepfer is Al Ready thinking. The future of Facebook he describes retrieves and coordinates information, like a butler are communicate with by typing or TA Lking as you might with a human one.

At the California Facebook headquarters, LeCun team members on the West Coast sat with Zuckerberg and company CTO Mike Schroepfer. The Facebook leader knows that the LeCun team is also building something to talk about, but Schroepfer is already thinking about how to use it. The future of Facebook that he describes can retrieve and integrate information as if it is communicating with a housekeeper, by typing or talking, and with the ability of a human steward to be similar.

"Can engage with a system, can really understand concepts and language at a much higher," says Schroepfer. He imagines being able to ask so you see a friend's baby snapshots but isn't his jokes, for example. "I think in the" a version of this is very realizable, "he says. As LeCun ' s systems achieve better reasoning and planning abilities, he expects the conversation to get less one-sided. Facebook might offer up information the IT thinks you ' d like and ask about you thought of it. "Eventually it's like this super-intelligent helper that's plugged in to all of the information streams in the world," says Schroepfer.

"You can use a system that really understands concepts and languages at a higher level," says Schroepfer. "For example, he imagines that the system will ask questions when he sees a friend's baby, and not when he sees his jokes," he said. "I think the feasibility is very high in the near future." "When the LeCun system had better reasoning and planning capabilities, he hoped the dialogue would not be so one-sided. Facebook may provide you with information you might like, and ask what you think. "Eventually it will be like a super-smart helper that connects all the information flow in the world," Schroepfer said. ”

The algorithms needed to power such interactions would also improve the systems Facebook uses to filter the posts and ads We see. And they could is vital to Facebook's ambitions to become much more than just a place to socialize. As Facebook begins to host articles and video on behalf of media and entertainment companies, for example, it'll need be Tter ways for people to manage information. Virtual assistants and other spinouts from LeCun's work could also help Facebook's more ambitious departures from its orig Inal business, such as the Oculus group working to make virtual reality into a mass-market technology.

The

Algorithms that support this connection will certainly help the Facebook system improve the filtering of posts and ads. Facebook's ambition is far more than a place to socialize, and these technologies are critical to this ambition. For example, Facebook needs to manage information better when it starts offering articles and videos to media and entertainment companies. The virtual assistants and other aspects of LeCun's work will also help Facebook navigate from the initial business to the distant, such as the Oculus Group is making virtual reality a huge market technology.

None of this would happen if the recent impressive results meet the fate of previous big ideas in artificial intelligence. Blooms of excitement around neural networks have withered twice already. But while complaining this other companies or researchers was over-hyping their work is one of the LeCun ' s favorite pastimes, He says there ' s enough circumstantial evidence to stand firm behind his own predictions the deep learning would deliver IM Pressive payoffs. The technology is still providing more accuracy and power in every area of AI where it has been applied, he says. New ideas is needed on how to apply it to language processing, but the Still-small field is expanding fast as Companie S and universities dedicate more people to it. "That'll accelerate progress," says LeCun.

If the artificial intelligence before a grand idea becomes reality, then none of this will happen. The excitement over the neural network has shrunk twice. But while complaining that other companies are over-publicizing their work, he says deep learning has paid off as much as his language, and the technology continues to provide more accuracy and energy in every area of application. New ideas are needed to apply them to natural language processing, but this still small area is expanding rapidly, and companies and universities are investing more people, "this will accelerate the process," LeCun said.

It's still not clear that deep learning can deliver anything like the information Butler Facebook envisions. And even if it can, it's hard-to-say how much the world really would benefit from it. But we are not having to wait for long to find out. LeCun guesses that virtual helpers with a mastery of language unprecedented for software would be available in just Five years. He expects that anyone who doubts deep learning's ability to master language would be proved wrong even sooner. "There is the same phenomenon that we were observing just before," he says. "Things is starting to work, and the people doing more classical techniques is not convinced. Within a year or both it'll be the end. "

Whether deep learning can achieve the information Facebook has predicted the butler's function is not yet clear, and even if so, how the world benefits from it is not sure, but it doesn't take long for us to find the results. LeCun guessing that the virtual assistant software with the language function will appear within the five years. He hopes to soon prove that people who doubt whether deep learning can master language skills are wrong. "We saw the same phenomenon 2012 years ago, and the algorithm is working, but people with traditional technical views have not been persuaded that a year or two will result," he said. ”

Teaching machines to understand us let the machine understand our belief in the three natural language learning and deep learning