Deep learning Open Source Picture Database Summary _ database

The preparation of the data is the necessary work before the training model, obviously it is also very time-consuming, so in the introductory phase we can use the existing open source image Library to quickly complete the preparatory work: Imagenet

Imagenet is an image database organized according to the WordNet hierarchy (currently only nouns), where each node of the hierarchy is depicted by hundreds of and thousands of images. Currently, each node in the database has more than 500 images on average. We hope that imagenet will become a useful resource for researchers, educators, students, and everyone who shares our passion for pictures.
Some features of imagenet:
Imagenet is the world's largest open source image library, up to Now (2017.5) imagenet a total of more than 14 million photos. These include more than 20,000 synset (s), Synset is a collection of synonyms, synsnet can be understood as imagenet collation of the label.
When it comes to wordnet hierarchies, mention what wordnet,wordnet is the open source lexicon of Princeton University, which can be understood as a dictionary. Each word (word) may have several different semantics, corresponding to different sense. Each of the different semantics (sense) may correspond to multiple words, such as topic and subject, which in some cases are synonymous, wordnet a cognitive Linguistics based English dictionary designed by psychologists, linguists and computer engineers at Princeton University. It is not the light that arranges the words in alphabetical order, and makes up a "network of words" according to the meaning of the word. WordNet has 3 main concepts Synset, Wordsense and Word. And Imagenet is the application of the concept of synset, but only imagenet currently only nouns.
Because of the copyright of the picture, the pictures in the imagenet are provided in the form of a URL, which means that imagenet only provides the picture and does not provide the image directly.

When we search for a synset in imagenet, we can see his hierarchy wordnet on the left and the download address for the URLs in download. Cifar

Cifar was collected and collated by Alex Krizhevsky, Vinod Nair and Geoffrey Hinton, and selected 60,000 of the 800,000 images in visual dictionary and divided them into CIFAR-10 and CIFAR-100.
The CIFAR-10 dataset contains 60,000 32*32 color images, with a total of 10 categories. There are 50,000 training images and 10,000 test images. The dataset is divided into 5 training blocks and a test block with 10,000 images per block. The test block contains 1000 randomly selected images from each class. The training blocks contain these images in a random order, but some training blocks may contain more images than other classes. The training block contains 5,000 images per class.
The CIFAR-100 DataSet contains 100 small classes, each containing 600 images, 500 training images and 100 test images. 100 classes are grouped into 20 large classes.
Mnist

Mnist in the field of deep learning, the famous dataset-mnist, almost all the introductory examples of in-depth tutorials are handwritten digit recognition, and the libraries they use are mnist. It's like when we learn a language that shows "Hello World".
The Mnist dataset contains a total of 70,000 samples, which are handwritten digital 0~9 and the sample size is 28*28.

Labeled Faces in the Wild

The labeled Faces in the Wild database collects more than 13,000 face images and contains more than 5,000 people. Each person is marked with other information besides the person's name, such as gender, age, etc.

Quick Draw

Quick Draw is a graffiti dataset published by Google that contains a collection of 50 million pictures, divided into 345 categories, which is actually like this:

So it looks like this dataset is still very boring, its release is actually originated from Google launched the Autodraw, this is a can make your graffiti into painting the Artificial intelligence technology tool, is this (Google is always doing something very fun):

Google has also published papers and blogs about the technology behind it. In fact, Autodraw's technology is based on Google's previous graffiti experiment "quick, draw!". Recently, Google released the project behind the data set, is mentioned in the Quick Draw data set. The project was posted to git at the same time, with a detailed description of the dataset in Git's address, briefly described below:

The original data is in the Ndjson file and is segmented by category, in the following format:

The dataset is stored in Ndjson files in the Google Cloud storage service. Refer to the list of files in the Cloud Console, sorted by dataset as follows:

Raw files (. Ndjson)
Simplified drawings files (. ndjson)
Binary files (. bin)
NumPy bitmap files (. npy)

Where the original files and strokes are all. Ndjson form Storage, both binary (. bin) and NumPy bitmap (. npy) files are provided.

Binary files (. bin)
We also provide a simplified custom binary format for painting and metadata that can be used for efficient compression and loading. Examples/binary_file_parser.py gives an example of how to load the file in Python.

NumPy bitmap (. npy)
All simplified paintings are also converted to 28x28 grayscale bitmaps, which are saved as numpy. npy format. The file can be loaded through the np.load () function. Ai-challneger

Ai-challneger is a competition initiated by the Innovation Workshop, which has 6 projects, each with a set of data sets, such as the scene classification project, so far provides three datasets, respectively, the training set (train), The validation set (valuation) and test set a (test_a) contain the picture 5w+,7k+,7k+, including the 80-class scene map, which supports the direct download of the original artwork.
Kaggle Cats vs. dogs

Cat and dog War data sets, the famous Kaggle competition data, a two classified data set, training, cat and dog pictures each 12500, test focus cat and dog pictures A total of 12500, supporting the original download.
Notmnist

The Notmnist data set does this by paying tribute to the mnist, which provides pictures from a to J that are letters, pictures of 28*28, and pictures that are not handwritten letters, but come from a variety of weird images on the web, such as the letter A:

The dataset provides two versions of which, in the large version, approximately 5.3W of each of the pictures, the total in the 53W, the data between the categories is more average. Pascal VOC

The PASCAL VOC challenge is a benchmark for the classification recognition and detection of visual objects, providing standard image annotation datasets and standard evaluation systems for detection algorithms and learning performance. The PASCAL VOC picture set consists of 20 catalogues: human, animal (bird, cat, ox, dog, horse, goat), means of transport (aircraft, bicycles, boats, buses, cars, motorcycles, trains); Indoors (bottles, chairs, dining tables, potted plants, sofas, televisions). The PASCAL VOC Challenge is no longer held after 2012 years, but its data sets are of good quality and fully annotated, and are ideal for testing algorithm performance.

COCO Common Objects Dataset

The

Coco DataSet is sponsored by Microsoft, the image annotation information not only has the category, the position information, but also has the semantic text description to the image, the Coco Data set open source makes the image segmentation semantic understanding has made the tremendous progress in the past two or three years, also almost becomes the image semantic understanding algorithm performance evaluation "standard" DataSet. Google's Open source Show & Tell generation model is tested on this dataset. The
currently included competition items are:
1. Target detection (COCO detection Challenge), contains two matches:
Output target border (using bounding box output), which is what we often say target detection (o Bject detection) The
requires the object to be separated from the image (object segmentation output), which is what we call the image semantic segmentation (semantic image segmentation)
2. Image annotations (COCO captioning Challenge)
Specifically, a sentence that accurately describes the information on the picture (producing image captions that are informative and accurate). How do I score this? Now it's a manual score.
3. Human key point detection (COCO KeyPoint Challenge)
The race requirement is to find out where the person is and then locate some key points in the body (the KeyPoint Challenge involves D Etecting people and localizing their keypoints).
Cityscapes

The cityscapes data scene consists of 50 different cities (mainly in Germany), the spring-summer and autumn three seasons of the day scene, with a large number of dynamic targets at different levels of the scene and diverse backgrounds. The scene does not include heavy rain and snow, because the scene needs to be treated with special techniques.
Image data is divided into 30 categories: In addition to 5000 frames (pixel level), marking a picture of the time control in the 1.5h or so, fine-labeled data divided into the following map training test set, not randomly divided, but to ensure that each partition of the dataset contains a variety of scenarios. Finally, there are 2975 pieces for training, 500 for validation and 1525 for testing. In addition, there are 20000 weak-labeled frames, used only for training, marking a picture to control within 7min.

Continuous update ...