convolutional neural Networks and Cifar-10:yann LeCun interview convolutional Nets and Cifar-10:an interview with Yann LeCun

Recently Kaggle hosted a competition on the CIFAR-10 dataset. The CIFAR-10 dataset consists of 60k 32x32 colour images in classes. This dataset is collected by Alex
Krizhevsky, Vinod Nair, and Geoffrey Hinton.

Many contestants used convolutional nets to tackle this competition. Some resulted in scores, beat human performance on this classification task. In the This blog series we'll interview three contestants and also a founding father of Convolutional Nets:yann LeCun.

EXAMPLE of CIFAR-10 DATASET

Yann LeCun

Yann LeCun is currently Director of AI in Facebook and a professor at NYU.

Which other scientists should is named and celebrated for the successes of convolutional nets?

Certainly, Kunihiko Fukushima's work on the Neo-cognitron is an inspiration. Although the early forms of convnets owed little to the Neo-cognitron, the version we settled on (with pooling layers) did .

A SCHEMATIC DIAGRAM illustrating the interconnections between LAYERS in the Neo-cognitron. From FUKUSHIMA K. (1980) Neocognitron:a self-organizing neural NETWORK MODEL for A mechanism of PATTERN recognition U Naffected by SHIFT in POSITION.

Can you recount an aha!-moment or theoretical breakthrough during your early the into convolutional nets?

Not really. It is the logical thing to do. I had been playing with multi-layer nets with local connections since 1982 or so (do not have the right learning algorithm Though. Backprop didn ' t exist). I started experimenting with gkfx weight nets while I am a postdoc in Toronto in 1988.

The reason for not trying earlier are simply that I didn ' t has the software nor the data. Once I arrived at Bell Labs, I had access to a large datasets and fast computers (for the time). So I could try full-size convnets, and it worked amazingly well (though it required 2 weeks of training).

What's your opinion on the recent popularity of convolutional nets for object recognition? Did you expect it?

Yes. I knew it had to happen. It is a matter of time until the datasets become large enough and the computers powerful enough for deep learning Algorit HMS to become better than human engineers at designing vision systems.

There was a symposium entitled ' Frontiers in Computer Vision ' at the MIT in August 2011. The title of my Talk is "5 years from now, everyone would learn their features (you might as well start now)". David Lowe (the inventor Ofsift) said the same thing.

A SLIDE from the talk LeCun Y. "5 years from now, EVERYONE would learn their FEATURES (you might as well START now)" .

Still, I was surprised by what fast the revolution happened and how much better convnets is, compared to other approaches. I would has expected the transition to being more gradual. Also, I would has expected unsupervised learning to play a greater role.

The character recognition model at T is more than a simple classifier, but a complete pipeline. Can you tell more about the implementation problems your team faced?

We had to implement our own program language and write our own compiler to build this. Leon Bottou and I had written a neural net simulator called SN, back in 1987/1988. It is a LISP interpreter with a numerical library (multidimensional arrays, neural net graphs ...). We used this on Bell Labs to develop the first convnets.

Then in the early-s, we wanted to use our code in products. Initially, we hired a team of developers to convert our Lisp code to C + +. But the resulting system could is improved easily (it wasn ' t a good platform for R/R). So Leon, Patrice Simard and I wrote a compiler for SN, which we used to develop the next generation OCR engine.

That system integrated a segmenter, a convnet, and a graphical the model on top. The whole thing is trained end to end.

The graphical model was called a "graph transformer network". It is conceptually similar to the what we are call a conditional random field, or a structured perceptron (which it predates) , but it allowed-non-linear scoring function (CRF and structured perceptrons can only has linear scoring functions).

The whole infrastructure is written in SN and compiled. This is the system, was deployed in ATM machines and check reading machines in 1996 and was reading ten to 20% of all t He checks in the US by the late ' s.

An ANIMATION SHOWING LENET 5 in ACTION. From "invariance and multiple characters with SDNN (multiple characters DEMO)".

In comparison with other methods, training convnets is pretty slow. How does deal with the trade-off between experimentation and increased model training times? What does a typical development iteration?

In my experience, the best large-scale learning systems have 2 or 3 weeks to train, regardless of the task, the Met Hod, the hardware, or the data.

I don ' t know if Convnets is "pretty slow". What's Compared to? They may slow to train and the alternative to "slow learning" are months of engineering efforts which doesn ' t work as W Ell in the end. Also, Convnets is actually pretty fast to run (after training).

In a real application, no one really cares how long it takes to train. But the people care a lot on how long it takes to run.

Which recent papers on convolutional nets is your most excited about? Any papers or ideas we should look out for?

There is lots and lots of ideas surrounding convnets and deep learning that has lived in relative obscurity for the last Years or so. No ones cared about it, and getting papers published is always a struggle. So, lots of ideas were never properly tried, never published, or were tried and published but soundly ignored and quickly Forgotten. Who remembers this first learning-based face detector that actually worked is a convolutional net (back in 1993, Eigh T years before Viola-jones)?

A figure with predictions from Vaillant R., Monrocq C., LeCun Y. (1993) "AnORIGINAL approach for the localisation of OBJECTS in IMAGES".

Today, it's really amazing to see so many young and bright people devoting so much creative energy to the topic and coming Up with new ideas and new applications. The Hardware/software infrastructure is getting better, and it's becoming possible to train large networks in a few hour s or a few days. So people can explore many more ideas that in the past.

One thing I ' m excited about are the idea of "spectral convolutional net". This is a paper at iclr by folks from my NYU labs about a generalization of convolutional nets the can is applied to Any graphs (regular convnets can is applied to 1 D, 2D or 3D arrays that can is seen as regular grids in terms of graph). There is practical issues, but it opens the door to many more applications of convnets to unstructured data.

MNIST DIGITS on A SPHERE. From BRUNA J., Zaremba W., Szlam A., LeCun Y. "Spectral NETWORKS and deeplocally CONNECTED NETWORKS on graphs< /c0> ".

I ' m very excited about the application of convnets (and recurrent nets) to natural language understanding (following the S Eminal work of Collobert and Weston).

Since the error rate of a human was estimated to be around 6, and Dr. Graham showed results of 4.47%, do you consider CIFA R-10 to be a solved problem?

It ' s a solved problem in the same sense as MNIST is a solved problem. But frankly, people is more interested inimagenet than in CIFAR-10 nowadays. In this sense, the CIFAR-10 is not a "real" problem. But it's not a bad benchmark for a new algorithm.

What would it take for Convnets to see a much wider adoption in the industry? Would training convnets and the software to set them up become less challenging?

What is talking about? Convnets is absolutely everywhere now (or about to being everywhere) in Industry:facebook, Google, Microsoft, IBM, Baidu, NE C, Twitter, Yahoo!....

That said, it's true that all of these companies has significant R & R resources and that training convnets can still B E challenging for smaller companies or companies that is less technically advanced.

It still requires quite of bit of experience and time investment to train a convnet if you don't have prior training. Soon However, there'll be a several simple-to-use open source packages with efficient back-ends for that.

is we close to the limit for convnets? Or could CIFAR-100 be "solved" next?

I Don ' t think it ' s a good test. ImageNet is a much better test.

Shallow nets can be trained the perform similarly to complex, well-engineered, deeper convolutional architectures. Does deep nets really need

Yes, deep nets need to is deep. Try to train a shallow net to emulate a deep convnet trained on ImageNet. Come. In theory, a deep net can is approximated by a shallow one. But on complex tasks, the shallow net'll has to be ridiculously large.

Most of your academic are highly practical in nature. Is this something you purposefully aim for, or was this a artefact of being employed by companies? Can you tell about the distinction between theory and practice?

Hey, I ' ve been in academia since 2003, and I ' m still a part-time professor at NYU. I do theory when it helps me understand things. Theory often help us understand what's possible and what's not possible. It helps suggest proper ways to do things.

But sometimes theory restricts our thinking. Some people won't work with Some models because the theory on them is too difficult. But often, a technique works well before the reasons for it working well is fully understood theoretically.

By restricting yourself to work on stuff you fully understand theoretically, you is condemned to using conceptually Simpl E methods.

Also, sometimes theory blinds us. For example, some people were dazzled by kernel methods because of the cute math that goes with it. But, as I ' ve said in the past, in the end, kernel machines is shallow networks that perform "glorified template matching" . There is nothing wrong with the (SVM is a great method), but it had dire limitations that we should all be aware of.

A SLIDE from lecun Y, learning hierarchies from invariant FEATURES

What's your opinion on a well-performing convnet without any theoretical justifications for why it should work so well? Do you generally favor performance over theory? Where do I place the balance?

I Don ' t think their is a choice to make between performance and theory. If There is performance, there'll be the theory to explain it.

Also, what kind of theory is we talking about? Is it a generalization bound? Convnets has a finite VC dimension, hence they is consistent and admit the classical VC bounds. What does you want? Do you want a tighter bound, like what's The Get for SVMs? No theoretical bound that I know of are tight enough to being useful in practice. So I really don ' t understand the point. Sure, generic VC bounds is atrociously non tight, but non-generic bounds (like for SVMs) is only slightly less atrocious ly non tight.

If you desire is convergence proofs (or guarantees), that's a little more complicated. The loss function of multi-layer nets is non convex, so the easy proofs that assume convexity was out the window. But we all know so in practice, a convnet would almost always converge to the same level of performance, regardless of th E starting point (if the initialization are done properly). There is theoretical evidence that there be lots and lots of equivalent local minima and a very small number of "bad" Loc Al minima. Hence convergence is rarely a problem.

What's your opinion on AI hype. Which practices do you think is detrimental to the field (of AI in general and specifically Convnets)?

AI hype is extremely dangerous. It killed various approaches to AI on least 4 times in the past. I keep calling out hype whenever I see it, whether it's from the press, from startups looking to investors, from large Co Mpanies looking for PR, or from academics looking for grants.

There is certainly quite a bit of hype around deep learning at the moment. I don ' t see a particularly high level of hype around convnets specifically. There is more hype around "cortical this", "spiking that", and "neuromorphic blah". Unlike many of these things, convnets actually yield good results on useful tasks and is widely deployed in industrial AP Plications.

Any interesting projects at Facebook involving Convnets so you could talk a little more about? Some Basic stats about the size?

Deepface:a convnet for the face recognition. There is also convnets for image tagging. They is big.

A figure describing the ARCHITECTURE from the PRESENTATION "Taigman Y., YANG M., Ranzato M., WOLF L. (A) DeepFace F OR Unconstrained face Recognition".

Recently posted about 4 types of serious researchers. How would are you label yourself?

I ' m a 3, with a bit of 1 and 4.

1. "People want to Explain/understand learning (and perhaps intelligence) at the fundamental/theoretical level.
2. People want to solve practical problems and has no interest in neuroscience.
3. People want to understand intelligence, build intelligent machines, and has a side interest in understanding how the Brain works.
4. People whose primary interest is to understand how the brain works, but feel they need to build computer models that Actua Lly work on order to does so. "

Anything wish to say to the top contestants in the CIFAR-10 challenge? Anything wish to say to (hobbyist) researchers studying convnets? Anything in general what wish to say about the Cifar Dataset/problem?

I ' m impressed by what much creativity and engineering knack went into this. It's nice that people has pushed the technique as far as it'll go on this dataset.

But it's going to get easier and easier for independent researchers and hobbyist to play with these things and apply them to larger datasets. I think the successor to CIFAR-10 should is imagenet-1k-128x128. This would is a version of the ImageNet classification task where the images has been normalized to 128x128 . I See several advantages:

1. The networks is small enough to being trainable in a reasonable amount for time on a high-end gamer rig;
2. The network get at the end can actually is used for useful application (like robot vision);
3. The network can be run with real time on embedded platforms, like smart phones or the NVIDIA Jetson TK1.

Predictions on IMAGENET. From "Krizhevsky A., Sutskever I., HINTON. G.E. (+) IMAGENET classification with deep convolutional neural NETWORKS".

The need to has large amounts of labeled data can be a problem. What are your opinion on nets trained on unlabeled data, or the automatic labeling of data through image search engines?

There is tasks like video understanding and natural language understanding where we is going to has to use unsupervised Learning. But these modalities has a temporal dimension that changes how we can approach the problem.

Clearly, we need to devise algorithms which can learn the structure of the perceptual world without being told the name of Everything. Many of us had been working on the For years (if not decades), but none of the US has a perfect solution.

What is your latest the focusing on?

There is answers to this question:

1. Projects I ' m personally involved in (enough that I would being co-author on the papers);
2. Projects that I set the stage for, encourage, and advise at the conceptual level, but which I am not invo Lved enough to is co-author on a paper.

A lot of (1) are at NYU and a lot of (2) are at Facebook.

The general areas is:

Unsupervised learning that discovers ' invariant ' features, the marriage of deep learning and structured prediction, the UN Ification of supervised and unsupervised learning, solving the problem of learning long-term dependencies, building Learni NG systems with short-term/scratchpad memory, learning plans and sequences of actions, different ways to optimize function s than to follow the gradient, the integration of representation learning with reasoning (read Leon Bottou ' s excellent pos Ition paper "From machine learning to Machine reasoning"), the use of learning to perform inference efficiently, and many Other topics.

from:http://blog.kaggle.com/2014/12/22/convolutional-nets-and-cifar-10-an-interview-with-yan-lecun/

convolutional neural Networks and Cifar-10:yann LeCun interview convolutional Nets and Cifar-10:an interview with Yann LeCun