Python scripts for migrating MySQL Data to MongoDB

Python scripts for migrating MySQL Data to MongoDB

MongoDB is a document database with a natural advantage in storing small files. As the business requirements change, you need to import the Row Records in the online MySQL database to the document records in MongoDB.

I. Scenario: migrate a table in an online MySQL database to MongoDB without changing the field.

Ii. Python module:
Use Python's torndb, pymongo, and time modules.
* Note: first install setup. py, pip, MySQLdb
Run the following command:
Pip install torndb
Pip install pymongo

Iii. Script content:
[Root ~] # Cat nmytomongo. py

#! /Usr/bin/env python
# FielName: mytomongo. py
# Author: xkops
# Coding: UTF-8
Import torndb, pymongo, time
# Connect to mysql database
Mysql = torndb. Connection (host = '192. 0.0.1 ', database = 'database', user = 'username', password = 'Password ')
# Connect to mongodb and obtain total lines in mysql
Mongo = pymongo. MongoClient ('mongodb: // ip'). database
Mongo. authenticate ('username', password = 'Password ')
Countlines = mysql. query ('select max (table_field) FROM table_name ')
Count = countlines [0] ['max (table_field) ']

# Count = 300
Print count

I = 0
J = 100
Start_time = time. time ()
# Select from mysql to insert mongodb by 100 lines.
For I in range (0, count, 100 ):
# Print a, B
# Print I
# Print 'select * FROM quiz_submission where quiz_submission_id> % d and quiz_submission_id <= % d' % (I, j)
Submission = mysql. query ('select * FROM table_name where table_field> % d and table_field <= % d' % (I, j ))
# Print submission
If submission:
# Collection_name like mysql table_name
Mongo. collection_name.insert_partition (submission)
Else:
I + = 100
J + = 100
Continue
I + = 100
J + = 100
End_time = time. time ()
Deltatime = end_time-start_time
Totalhour = int (deltatime/3600)
Totalminute = int (deltatime-totalhour * 3600)/60)
Totalsecond = int (deltatime-totalhour * 3600-totalminute * 60)
# Print migrate data total time consuming.
Print "Data Migrate Finished, Total Time Consuming: % d Hour % d Minute % d Seconds" % (totalhour, totalminute, totalsecond)

* Note: Change the database address, user, password, database name, table name, and field name in the above Code as needed.

4. Execute the migration script:
[Root ~] # Python nmytomongo. py &>/tmp/migratelog.txt &
After the script is executed, view the time consumed for migration of/tmp/migratelog.txt data.

For more MongoDB tutorials, see the following:

CentOS compilation and installation of php extensions for MongoDB and mongoDB

CentOS 6 install MongoDB and server configuration using yum

Install MongoDB2.4.3 in Ubuntu 13.04

MongoDB beginners must read (both concepts and practices)

MongoDB Installation Guide for Ubunu 14.04

MongoDB authoritative Guide (The Definitive Guide) in English [PDF]

Nagios monitoring MongoDB sharded cluster service practice

Build MongoDB Service Based on CentOS 6.5 Operating System

MongoDB details: click here
MongoDB: click here

This article permanently updates the link address: