Reference article: Mongo Hadoop connector https://github.com/mongodb/mongo-hadoopwikihttps://github.com/mongodb/mongo-hadoop/ Wikippthttp://www.slideshare.net/mongodb/hadoop-webinar?from_embed_lead_cta=true&tracking_id= 167b463f3b941754 Tutorial http://docs.mongodb.org/ecosystem/tools/hadoop/
The steps for deployment are simple and are summarized as follows: 1: Download wget https://github.com/mongodb/mongo-hadoop/archive/r1.4.0-rc0.tar.gz
2: Compile the extract after compiling and follow the documentation instructions./gradlew Jarafter a successful compilation, we need to find three jar packages, core/build/libs and hive/build/libs below are two jar packages as follows:Mongo-hadoop-core-1.4-rc0.jarMongo-hadoop-hive-1.4-rc0.jarThere is also a jar package to go under the official web, as follows:http://mongodb.github.io/mongo-java-driver/Mongo-java-driver-2.13.2.jar 3: Deployment and Configurationcopy each of the three jar packages to your locationfor each node:$HADOOP _home/share/hadoop/mapreduce$HADOOP _home/lib$HIVE _home/libOn Hive-site.xml, you'll be appended to the Hive.aux.jars.path configuration.Mongo-hadoop-hive-1.4-rc0.jar File
4: The table statement is as follows:CREATE TABLE Test
(
ID string,
Title String,
Desc string,
UID String,
CID String,
Attach Struct<atype:string,itemid:string,bizid:string>
Weight string,
Status String,
Loc Struct<type:string,coordinates:string>
Created string,
Modified string
)
Stored by ' Com.mongodb.hadoop.hive.MongoStorageHandler '
With Serdeproperties (' mongo.columns.mapping ' = ' {"id": "_id", "title": "title", "desc": "desc", "UID": "UID", "CID": "CID" , "Attach.atype": "Attach.atype", "Attach.itemid": "Attach.itemid", "Attach
. Bizid ":" Attach.bizid "," Weight ":" Weight "," status ":" Status "," Loc.type ":" Loc.type "," Loc.coordinates ":" Loc.coordinates "," created ":" Created "," Modified ":" Modified "} ')
Tblproperties (' mongo.uri ' = ' mongodb://your_mongo_ip:27017/database_name.collection_name ');
Note that inside the mapping, you need to match the fields in the MONGO Bson with the fields in the Hive table.
5:hueIf you use Hue, update its hive-site.xml file synchronously, and restart both the Hivemetastore and HiveServer2 processes. then restart Hue. You can now query the Mongdb table in hue.
6: Special attentionAfter you've built the table, if you're connected to the main library of production, and you want to modify the table, do not drop the MongoDB table in hive directly. I used to build the table when the path is the main library. After the drop of Hive's MONGO table, the table in MongoDB was also drop. The mongo-hadoop-connector table created in hive actually maps the MONGO data to the hive table so that it can be queried using hive's SQL, but in fact the data is still in MongoDB. So if you want to delete a table, divide the jar file by the disconnection and then delete the Hive table, or the data inside the MONGO will be deleted directly. Of course, OPS is not doing well. MongoDB Rights Management is also a factor.
Conclusion:the company's production environment is hadoop2.5.1+spark1.4+hive0.13.1, and Mongo-hadoop-core-1.4-rc0.jar is basically a good support for this environment (just a few simple SQL statements have been tested). For security reasons:(1) When using this plugin, please connect the MONGO from the library. (2) When deleting the MONGO table in hive, remove the jar package configuration in Hive-site.xml, and after disconnecting the Hive table from the MONGO, be careful! If you keep driving and drop directly, the table data in the MONGO will be drop. The most important thing is that your ops should do the right configuration of MongoDB in advance and give read-only permission to prevent misoperation. (3) in general, as long as the configuration is good, this plug-in can be very good to save the ETL work, so that hive can be directly above the MONGO to perform hivesql query, must pay attention to data security.
Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.
Experience and lessons on using MongoDB Connector for Hadoop
