Python crawler supplement, installing MongoDB and Redis databases on the master node that machine is a non-apt approach

It is not only cumbersome but also very redundant to find data crawled by crawlers that are stored in a table structure.

Try this non-relational database to try out how the store works.

I'm not going to compare it with Redis, because he's a memory database, and his area of expertise should be a statistical collation of caches and small amounts of data.

(There's another big guy in this memcache), and Redis later improves efficiency with other applications.

The main comparison here is the poor performance of MongoDB and MySQL, on the specific refers to such a complex relationship network application Environment!!

Apt-cache depends # (package understanding using dependencies)
Apt-cache Rdepends # (package understands a specific dependency? #When looking at which packages are dependent on this package ...)
sudo apt-get BUILD-DEP # (compilation Environment related to package installation)
Apt-get Source # (The package downloads the source code for the bundle)
sudo apt-get clean && sudo apt-get AutoClean # cleans up the archive of downloaded files && cleans only outdated packages
sudo apt-get check #Check for broken dependencies











Add, if you install the package suddenly interrupted error,





sudo rm / var / lib / dpkg / updates / *





The main reason should be the / var / lib / dpkg / updates folder inside the data has errors, so that the software update program errors, so you have to remove them completely,





This command will re-establish this information via sudo apt-get update, using sudo apt-get upgrade to update the details of the installed software on your computer, according to the software's detailed update software to the latest version.





sudo apt-get update #Update source
sudo apt-get upgrade #Update installed packages

, unlike sudo apt-get Dist-upgrade # This is an upgrade system

Dpkg and Apt-get Accidental damage

Then you have to do it first:

sudo rm/var/cache/apt/archives/lock
sudo rm/var/lib/dpkg/lock

Do the above update again

First install MongoDB:

Refer to MongoDB website Installation documentation: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

This has the method of how to uninstall the old version and install it in apt-get way:

I strongly recommend that you install it in the APT way!!!!

I strongly recommend that you install it in the APT way!!!!

I strongly recommend that you install it in the APT way!!!!

Don't learn me, the bitter lesson ... Just follow the apt tutorial of MongoDB website to install!!!!

I did not choose apt's method, because I am novice, apt's method will divide the file to each system folder, I do not understand

And I just want to use my own set of own users using MongoDB do not use root

So I'm still installing it as a unpacking package.

After I downloaded the package, I found altogether no conf file. It's all compiled program files.

1
Curl-o https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.0.5.tgz
2
TAR-ZXVF mongodb-linux-x86_64-3.0.5.tgz
3
Mkdir-p mongodb-3.0.5/
Cp-r-N mongodb-linux-x86_64-3.0.5/~/mongodb-3.0.5/
4
Export Path=<mongodb-install-directory>/bin: $PATH

Running

1
Mkdir-p/data/db
2
Make sure you have permission to read and write

3
Start
Mongod--dbpath <path to Data directory>

sudo chmod-r 777/home/luis/mongodb-3.0.5/data/db/

And then there's some auxiliary work.

To modify the maximum number of connections:

Modify the/etc/security/limits.conf configuration file.
Use command: sudo gedit/etc/security/limits.conf
Add in File
* Soft Nofile 3000
* Hard Nofile 20000
Root Soft Nofile 3000
Root Hard Nofile 20000
* Indicates that the configuration is valid for all users, and the root user has to add two extra lines.
The hard limit is usually the number of files that can be opened at the same time based on the system's Hardware resource status (mainly system memory), and the soft limit is further limited on this basis. So the number of soft limits is lower than the hard limit.
Nofile represents max number of open files
Restart the computer, and then use the ULIMIT-A command to view:
Open files (-N) 3000
has taken effect. Now start MongoDB server again, problem solved

Reboot to take effect

Then set the boot up

Create a new script file in the/etc/init.d/directory MongoDB

#!/bin/sh  
   
### BEGIN INIT INFO  
# Provides:     mongodb  
# Required-Start:  
# Required-Stop:  
# Default-Start:        2 3 4 5  
# Default-Stop:         0 1 6  
# Short-Description: mongodb  
# Description: mongo db server  
### END INIT INFO  
   
. /lib/lsb/init-functions  
   
PROGRAM=/home/luis/mongodb-3.0.5/bin/mongod
MONGOPID=`ps -ef | grep 'mongod' | grep -v grep | awk '{print $2}'`  
   
test -x $PROGRAM || exit 0  
   
case "$1" in  
  start) 
     ulimit -n 3000  
     log_begin_msg "Starting MongoDB server"   
     $PROGRAM --fork --quiet -journal -maxConns=2400 -rest --dbpath /home/luis/mongodb-3.0.5/data/db --logpath /home/luis/mongodb-3.0.5/data/db/journal/mongodb.log
     log_end_msg 0  
     ;;  
  stop)  
     log_begin_msg "Stopping MongoDB server"   
     if [ ! -z "$MONGOPID" ]; then   
        kill -15 $MONGOPID  
     fi  
     log_end_msg 0  
     ;;  
  status)  
     ;;  
  *)  
     log_success_msg "Usage: /etc/init.d/mongodb {start|stop|status}"   
     exit 1  
esac  
   
exit 0

Use the sudo chmod +x/etc/init.d/mongodb command to allow the script to be executed
Then run the following command to register the boot script:
UPDATE-RC.D MongoDB Defaults

You can also delete by update-rc.d-f MongoDB remove
Reboot, via Ps-def | grep Mongod can view the self-initiated service process, and can then close/start the service with the following command
sudo service MongoDB stop
sudo service MongoDB start

Client Logon server
Start the log as above, the server started as above, and now we test the servers in another terminal is normal.
Enter/usr/local/mongodb-linux-x86_64-2.0.2/bin, execute./mongo
Appear
MongoDB Shell version:2.0.2
Connecting To:test
Execute Db.foo.save ({1: "Hello World"})
Then find Db.foo.find ();
See {"_id": ObjectId ("4e4b395986738efa2d0718b9"), "1": "Hello World"}
Execute here to congratulate you, successfully installed MongoDB


You can also connect to a remote MongoDB server in this way, with a default port of 27017, such as
./mongo 192.168.30.25
Create a database
If you do not have a mydb database, use the command in the client:
Use MyDB
The MyDB database is created and the current database is switched to MyDB.
Show DBS does not display the database name at this time. Use the Db.stats () command to check the current database state.
Standard inspection Process
1. First check the Ulimit-a
View whether open files (-N) is a set value
If it is set in the boot script, it should be checked by the following 4 steps. This step can be ignored.


2.ps-def | grep Mongod
See if the service starts
3.cd/data/db/journal/
Cat Mongdb.log
See if the server is correct
4. Enter http://192.168.1.199:28017
See if the server is up and running
5. Enter/usr/mongodb/bin, execute./mongo
See if you can log in

To install Redis:

Reference official website: http://redis.io/download

Installation

$ wget http://download.redis.io/releases/redis-3.0.3.tar.gz
$ tar xzf redis-3.0.3.tar.gz
$ CD redis-3.0.3
$ make

The binaries that is now compiled is available in the SRC directory. Run Redis with:
$ src/redis-server

You can interact with Redis using the built-in client:
$ src/redis-cli
Redis> set Foo Bar
Ok
Redis> get foo
"Bar"

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Python crawler supplement, installing MongoDB and Redis databases on the master node that machine is a non-apt approach