MongoDb gridfs-ngnix file storage solution, mongodbgridfs

MongoDb gridfs-ngnix file storage solution, mongodbgridfs

In the development of various system application servers, we often encounter file storage problems. Traditional DBMS file stream storage is a common disk file system. Today, let's take a look at the NoSQL database MongoDb-based storage solution. Take CentOS 6.5, MongoDb 2.6.3, Nginx-1.4.7 as an example, you need to understand common Linux commands.
Let's review the internal file structure of MongoDb.

{"_ Id": ObjectId ("4f4608844f9b855c6c35e298"), // unique id, which can be a user-defined type "filename": "CPU.txt", // file name "length ": 778, // file length "chunkSize": 262144, // chunk size "uploadDate": ISODate ("2012-02-23T09: 36: 04.593Z"), // upload time "md5 ": "e2c789b036cfb3b848ae39a24e795ca6", // md5 value of the file "contentType": "text/plain" // MIME type of the file "meta": null // other information of the file, the key "meta" is not used by default. You can define it as any BSON object}

The chunks in the corresponding fs. chunks are as follows:

{"_ Id": ObjectId ("4f4608844f9b855c6c35e299"), // chunk id "files_id": ObjectId ("delimiter"), // File id, corresponding to fs. objects in files, equivalent to fs. the foreign key "n" of the files set: 0, // The chunk block of the file. If the file is larger than chunksize, it is divided into multiple chunk blocks "data ": binData (0, "QGV... ") // binary data of the file. The specific content is omitted here}

When the file is stored in GridFS, if the file is larger than chunksize, the file is divided into multiple chunks and then saved to fs. chunks, and finally store the file information to fs. files.

When reading a file, the first data query condition is in fs. files to find a suitable record, get the value of "_ id", and then use this value to fs. in the chunks, find all the chunks whose "files_id" is "_ id" and sort them by "n". Then, read the content of the "data" object in the chunk in sequence and restore it to the original file.

Install and configure

1. Install mongoDb

Add MongoDB Repository. For details about vim, refer to VIM.

  
vim /etc/yum.repos.d/mongodb.repo

For 64bit

  
[mongodb]
  
name=MongoDB Repository
  
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
  
gpgcheck=0
  
enabled=1

32bit system:

  
[mongodb]
  
name=MongoDB Repository
  
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686/
  
gpgcheck=0
  
enabled=1

After installation, the system prompts Y/N:

  
yum install mongo-10gen mongo-10gen-server

Start:

  
service mongod start

View status

  
service mongod status

Stop

  
service mongod stop

For more information about version 3.0 and later, see the official website.

2. Install nginx and nginx-gridfs

Dependent libraries and tools

  
# yum -y install pcre-devel openssl-devel zlib-devel
  
# yum -y install gcc gcc-c++

Download nginx-gridfs source code

  
# git clone https://github.com/mdirolf/nginx-gridfs.git
  
# cd nginx-gridfs
  
# git checkout v0.8
  
# git submodule init
  
# git submodule update

Download the nginx source code and compile and install it. (High Version support is not good)

  
# wget http://nginx.org/download/nginx-1.4.7.tar.gz
  
# tar zxvf nginx-1.4.7.tar.gz
  
# cd nginx-1.4.7
  
# ./configure --with-openssl=/usr/include/openssl --add-module=../nginx-gridfs/
  
# make -j8 && make install –j8

Note that the blue character is configured to the nginx-gridfs path.

3. Configure nginx-gridfs

Vim/usr/local/nginx/conf/nginx. conf

Add a location node to the server node

Location/img /{
Gridfs testdb
Field = filename
Type = string;
Mongo 192.168.0.159: 27017;
}

Location/files /{
Gridfs testdb
Field = _ id
Type = objectid;
Mongo 192.168.0.159: 27017;
}

Here, our mongo service is at IP address 192.168.0.159.
If no field is specified, the default value is MongoDB's auto-increment ID and the type is int.

Configuration parameters:

Gridfs: keyword used by nginx to identify plug-ins
Testdb: db name
[Root_collection]: select collection, for example, root_collection = blog. mongod will find the blog. files and blog. chunks. The default value is fs.
[Field]: query a field. Make sure that the field name is in mongdb. The parameter _ id and filename are supported, which can be omitted. The default value is _ id.
[Type]: indicates the Data type of the field. objectid, int, string are supported, which can be omitted. The default value is int.
[User]: user name, which can be omitted
[Pass]: password, which can be omitted
Mongo: mongodb url

Start the nginx Service

  
# /usr/local/nginx/sbin/nginx

Possible causes:
Nginx [emerg]: bind () to 0.0.0.0: 80 failed (98: Address already in use)

You can use commands to close the program that uses port 80.

  
sudo fuser -k 80/tcp

Simple Test

Upload a file using native command lines

  
mongofiles put 937910.jpg --local ~/937910_100.jpg --host 192.168.0.159 --port 27017 --db testdb --type jpg

92.1610.jpg is an image file we downloaded in advance. Note that the collection is not specified. The default value is fs.

Install the robotium management tool from http://www.robow..org/to view the uploaded file.

Int nFileLen = fileUploadModel. fileBytes. length; required gridfsset=fssetting = new required gridfssettings () {Root = CollectionName}; required gridfs fs = new required gridfs (Response server, required databasename, fsSetting ); // You must manually set the upload time when calling the Write, WriteByte, and WriteLine functions. // Add additional information using Metadata. Optional gridfscreateoptions option = new required gridfscreateoptions (); option. id = ObjectId. generateNewId (); var currentDate = DateT Ime. Now; option. UploadDate = currentDate; option. Aliases = alias; BsonDocument doc = new BsonDocument (); // store additional document information if (fileUploadModel. DocExtraInfo! = Null & fileUploadModel. DocExtraInfo. Count> 0) {foreach (var obj in fileUploadModel. DocExtraInfo) {if (! Doc. elements. any (p => p. name = obj. key) {doc. add (obj. key, obj. value) ;}} option. metadata = doc; // create a file and store the data using (using gridfsstream gfs = fs. create (fileUploadModel. fileName, option) {gfs. write (fileUploadModel. fileBytes, 0, nFileLen); gfs. close ();} log. errorFormat ("attachment ID: {0} file name: {1} uploaded successfully", alias, fileUploadModel. fileName); return option. id. toString ();

Note: currently, gridfs-ngnix does not support GUID for the _ id type. For details about ObjectId, refer to the official website, for example:

Author: Petter Liu
Source: http://www.cnblogs.com/wintersun/
The copyright of this article is shared by the author and the blog Park. You are welcome to repost this article. However, you must retain this statement without the author's consent and provide a clear link to the original article on the article page. Otherwise, you will be held legally liable.
This article is also published in Petter Liu Blog, my independent Blog.