20171124_python study six weeks five lessons (November 24)

Today's Mission:

13.1 NoSQL Introduction
13.2 Redis Service Setup
13.3 Redis Connection Pool
13.4 Redis Pipeline

Notes:

1. Python Operation NoSQL Database

NoSQL, which refers to non-relational databases. With the rise of internet web2.0 website, the traditional relational database in coping with web2.0 website, especially the web2.0 pure dynamic website of ultra-large-scale and high-concurrency SNS type, has been unable to overcome, exposing a lot of difficult problems, and the non-relational database has been developed very rapidly because of its own characteristics. NoSQL databases are created to address the challenges of multiple data types in large-scale data sets, especially big data application challenges.

Although the popularity of NoSQL and the fire took just a few years, it is undeniable that the second generation of movement has now begun. Although the early stack code can only be considered an experiment, the present system is more mature and stable. But now there is a grim fact: technology is getting more mature--so much so that the good NoSQL data store has to be rewritten, and a few think it's called version 2.0. The tool can build a fast, extensible repository for big data.

The difference between a non-relational database and a relational database:

Benefits of non-relational databases: 1. Performance NoSQL is based on key-value pairs, which can be imagined as the corresponding relationship between the primary key and the value in the table, and does not need to be parsed by the SQL layer, so the performance is very high. 2. Extensibility is also due to the fact that there is no coupling between data based on key-value pairs, so it is easy to scale horizontally. Benefits of Relational databases: 1. Complex queries can easily use SQL statements to make very complex data queries between a table and multiple tables. 2. Transactional support enables data access requirements with high security performance. For these two types of databases, the other's advantage is their own weakness, and vice versa. But in recent years both of these databases have evolved in a different direction. For example, NoSQL databases are slowly beginning to have some of the complex query functions of SQL database, such as Couchbase's index and MONGO's complex queries. Support for transactions can also use some system-level atomic operations to implement methods such as optimistic locking to curve the salvation. SQL database also began to evolve slowly, such as the implementation of Handlersocker technology, can be implemented on the SQL layer of MySQL penetration, in a nosql way to access the database, performance can be achieved or even beyond the NoSQL database. scalability, such as Percona Server, enables the implementation of a cluster that is not centralized. Although the Poles are beginning to evolve another pole because of their weaknesses, the increase in these features will also weaken the advantages they would otherwise have, such as the increase in index on the couchbase gradually reduces the read and write performance of the database. So how to build a system's short-and long-term storage strategy, using their strengths is an important issue that architects need to think about.

1. Python Operation Redis

The concept of Redis:

Redis is a key-value storage system. Similar to memcached, it supports storing more value types, including string (string), list (linked list), set (set), Zset (sorted set-ordered collection), and hash (hash type). These data types support Push/pop, Add/remove, and intersection-set and difference sets, and richer operations, and these operations are atomic. Based on this, Redis supports sorting in a variety of different ways. As with memcached, data is cached in memory to ensure efficiency. The difference is that Redis periodically writes the updated data to disk or writes the modified operation to the appended record file, and Master-slave (Master-Slave) synchronization is implemented on this basis.

Redis is a high-performance Key-value database. The emergence of Redis, to a large extent, compensates for the lack of memcached such key/value storage, in some cases can be a good complement to the relational database. It provides clients such as Java,c/c++,c#,php,javascript,perl,object-c,python,ruby,erlang, which is convenient to use.

Redis supports master-slave synchronization. Data can be synchronized from the primary server to any number of slave servers, from the server to the primary server that is associated with other slave servers. This enables Redis to perform single-layer tree replication. You can write to the data intentionally or unintentionally. Because of the full implementation of the publish/subscribe mechanism, you can subscribe to a channel and receive a complete message release record from the master server when the tree is synchronized anywhere from the database. Synchronization is helpful for the scalability and data redundancy of read operations.

Installation of Redis

Redis is typically installed on a Linux system with the following installation steps:

#cd/USR/LOCAL/SRC

#wget http://download.redis.io/releases/redis-3.0.1.tar.gz

#tar Xzf redis-3.0.1.tar.gz

#cd redis-3.0.1

#make

#src/redis-server &

Check if Redis starts properly

Ps–ef |grep Redis

NETSTAT–LNP |grep 6379

Client Installing Redis

Pip Install Redis

1, simple operation:

Redis is stored in the form of key-value, so when we operate. First we instantiate the IP and publish port of the Redis host as an object R, then execute set (' name ', ' I love you! '), so we store a key in memory as Shang, value ' I love you! ' The item. We can understand that {' Shang ': ' I Love you! '}, when we want to read, keys () is to obtain more than the key value.

#-*-Coding:utf-8-*-

# @Time: 2017/9/17 19:19

# @Author: Lingxiangxiang

# @File: demon1.py

Import Redis

R = Redis. Redis (host= "192.168.48.128", port=6379)

R.set ("Shang", "I Love you!")

Print (R.get ("Shang"))

Print (R.keys ())

# Print (dir (r))

Results:

I Love you!

[' Shang ', ' ajing ']

2. Connection pooling

Redis-py uses connection pool to manage all connections to a Redis server, avoiding the overhead of each establishment and release of the connection. By default, each Redis instance maintains its own pool of connections. You can create a connection pool directly, and then as a parameter Redis, you can implement multiple Redis instances to share a single connection pool

#!/usr/bin/env python

#-*-Coding:utf-8-*-

# @Time: 2017/9/17 23:54

# @Author: Lingxiangxiang

# @File: demon2.py

Import Redis

Pool = Redis. ConnectionPool (host= "192.168.48.128")

R = Redis. Redis (Connection_pool=pool)

R.set ("name", "Lingxiangxiang")

Print (R.get ("name"))

Results:

Lingxiangxiang

Pipeline

Redis-py The default is to create each request (Connection pool request connection) and disconnect (return connection pool) One connection operation, if you want to specify more than one command in a single request, you can use Pipline to implement a single request to specify multiple commands, and by default Pipline is an atomic operation. Reduced power consumption

Redis is a CS-mode TCP server that uses a request-response protocol similar to HTTP. A client can initiate multiple request commands from a single socket connection. After each request command is issued, the client usually blocks and waits for the Redis service to process, and the result is returned to the client via a response message after Redis finishes processing the request. The basic communication process is as follows:

CLIENT:INCR X

Server:1

CLIENT:INCR X

Server:2

CLIENT:INCR X

Server:3

CLIENT:INCR X

Server:4

Basically four commands require 8 TCP messages to complete. Due to the network latency of the communication, the packet transfer time between the client and server takes 0.125 seconds. Then the above four commands 8 messages will take at least 1 seconds to complete. This way, even though Redis can handle 100 commands per second, our client can only issue four commands in a second. This shows that the processing power of Redis is not fully utilized. In addition to the ability to handle multiple key commands with a single command such as Mget,mset, we can also use pipeline to send out multiple commands from the client, without waiting for the response of a single command to return. The Redis server then processes multiple commands and packages the results of multiple commands back to the client. The communication process is as follows:

CLIENT:INCR X

CLIENT:INCR X

CLIENT:INCR X

CLIENT:INCR X

Server:1

Server:2

Server:3

Server:4

Assume that the TCP message is not split because it is too long. Perhaps two TCP messages can complete four commands, the client can put four commands in a TCP message sent together, the server can put the processing results of four commands to a TCP message return. By pipeline mode when there is a large amount of operation. We can save a lot of time originally wasted on network latency. Note that the pipeline Package command is sent, and Redis must cache all command processing results before all commands are processed. The more commands are packaged, the more memory is consumed by the cache. So the more you pack the more commands the better. Specific how much appropriate needs to be tested according to specific circumstances

#!/usr/bin/env python

#-*-Coding:utf-8-*-

# @Time: 2017/10/9 22:45

# @Author: Lingxiangxiang

# @File: demon4.py

Import datetime

Import Redis

def withpipe (R):

Pipe = R.pipeline (transaction=true)

For I in xrange (1, 1000):

Key = "Test1" +str (i)

Value = "Test1" + str (i)

Pipe.set (key, value)

Pipe.execute ()

def withoutpipe (R):

# pipe = R.pipeline (transaction=true)

For I in xrange (1, 1000):

Key = "Test1" + str (i)

Value = "Test1" + str (i)

R.set (key, value)

if __name__ = = "__main__":

Pool = Redis. ConnectionPool (host= "192.168.48.131", port=6379, Db=0)

R1 = Redis. Redis (Connection_pool=pool)

r2 = Redis. Redis (Connection_pool=pool)

Start = Datetime.datetime.now ()

# print (start)

Withpipe (R1)

End = Datetime.datetime.now ()

# Print ((End-start). microseconds)

# Print (End-start)

T_time = (end-start). microseconds

Print ("Withpipe time is: {0}". Format (T_time))

Start = Datetime.datetime.now ()

Withoutpipe (R2)

End = Datetime.datetime.now ()

T_time = (end-start). microseconds

Print ("Withoutpipe time is: {0}". Format (T_time))

Results:

Withpipe Time is:28000

Withoutpipe Time is:253000

Reids currently offers 5 data types: String type, list type, set set type, ordered collection type, hash type, and below we will detail their use.

String manipulation

A string in Redis is stored in memory by a key corresponding to a value. Such as:

R.set ("name", "Lingxiangxiang")

How to use Set:

Set (name, value, Ex=none, Px=none, Nx=false, Xx=false)

EX, expiry time (seconds)

PX, Expiration Time (ms)

NX, if set to true, the current set operation executes only if name does not exist, with SETNX (name, value)

XX, if set to true, the current set operation only executes "if the name exists.

Get (name) get value

Print (R.get ("name"))

Mset () Batch setting value

R.mset (name1= "Shang", name2= "Ling")

R.mset ({"Name3": "Kong", "name4": "gu"})

Mget (keys, *args) bulk fetch values

Print (R.mget ("name1", "name2", "Name3", "Name4"))

Append (name, value)

R.append ("name", "Lisi")

Print (R.get ("name"))

List operation

The list in Redis is stored in memory according to a list of name

Lpush (name,values)

# add elements to the list in name, and each new element is added to the leftmost

R.lpush ("List_name", 2)

R.lpush ("List_name", 3,4,5) #保存在列表中的顺序为5, 4,3,2

Rpush (name,values)

#同lpush, but each new element is added to the far right of the list

Lpushx (Name,value)

#在name对应的list中添加元素, the value is added to the leftmost side of the list only if name already exists

Rpushx (Name,value)

#在name对应的list中添加元素, the value is added to the far right of the list only if name already exists

Llen (name)

# name corresponds to the number of list elements

Print (R.llen ("List_name")),

Linsert (Name,where,refvalue,value): Inserts a new value before and after a value in the list corresponding to name

#参数:

# Name,redis's name

# Where,before or after

# Refvalue, the benchmark value, that is: Insert data before and after it

# value, the data to be inserted

R.lset (Name,index,value) : Re-assigns a value to one of the index locations in the list that corresponds to name.

Parameters

# Name,redis's name

# index position of Index,list

# value, values to set

R.lrem (name,value,num): Deletes the specified value in the list corresponding to name

Parameters

# Name,redis's name

# value, values to remove

# num, num=0, removes all specified values from the list;

# num=2, from front to back, delete 2;

# num=-2, from back forward, delete 2

Lpop (name) : Gets the first element on the left side of the list corresponding to name and removes it from the list, and the return value removes the value of that element

#扩展: Rpop (name) indicates right-to-left operation

lindex (Name,index) : Takes a list element from an index in the list corresponding to name

Lrange (name,start,end) : Gets the data in the list of the name corresponding to the Shard

#!/usr/bin/env python

#-*-Coding:utf-8-*-

# @Time: 2017/10/10 21:20

# @Author: Lingxiangxiang

# @File: demon6.py

Import Redis

Pool = Redis. ConnectionPool (host= "192.168.48.131", port=6379, Db=0)

R = Redis. Redis (Connection_pool=pool)

# Lpush add an element to the left of list

# Rpush add an element to the right of list

R.lpush ("List1", "Test1")

R.rpush ("List1", "Ling")

R.lpush ("List1", "test2")

R.lpush ("List1", "test3")

R.lpush ("List1", "test2")

R.lpush ("List1", 2, 3, 4)

R.lpush ("List1", "test2")

Print (R.lrange ("List1", 0,-1))

# The final list result is [4, 3, 2, "Test3", "Test2", "Test1", "Ling"]

# add an element in an intermediate position

# Linsert (Name,where,refvalue,value)

# Name represents the value of the list corresponding to the key

# where after before

# Refvalue List one of the elements in

# value What is the value you want to add?

R.linsert ("List1", "after", "Test2", "Hello")

Print (R.lrange ("List1", 0,-1))

R.lset ("List1", 2, "World")

# LSet change the value of an element

# R.lset (Name,index,value)

Print (R.lrange ("List1", 0,-1))

Print (R.lindex ("List1", 2))

# lindex View the value of a list's underlying

Print (R.lpop ("List1"))

# Lpop deletes an element from the leftmost list, returning the value of the deleted element

Print (R.lrange ("List1", 0,-1))

R.lrem ("List1", "World")

# R.lrem (name,value,num):

# num , num=0, deletes all the specified values from the list;

# num=2, before and after, delete 2;

# num=-2, from back forward, remove 2

Print (R.lrange ("List1", 0,-1))

Results:

[' Test2 ', ' 4 ', ' 3 ', ' 2 ', ' test2 ', ' test3 ', ' test2 ', ' test1 ', ' Ling ']

[' Test2 ', ' Hello ', ' 4 ', ' 3 ', ' 2 ', ' test2 ', ' test3 ', ' test2 ', ' test1 ', ' Ling ']

[' Test2 ', ' Hello ', ' World ', ' 3 ', ' 2 ', ' test2 ', ' test3 ', ' test2 ', ' test1 ', ' Ling ']

World

Test2

[' Hello ', ' World ', ' 3 ', ' 2 ', ' test2 ', ' test3 ', ' test2 ', ' test1 ', ' Ling ']

[' Hello ', ' 3 ', ' 2 ', ' test2 ', ' test3 ', ' test2 ', ' test1 ', ' Ling ']

Set operation

Set set is a list that is not duplicated

Sadd (name,values)

#给name对应的集合中添加元素

R.sadd ("Set_name", "AA")

R.sadd ("Set_name", "AA", "BB")

Smembers (name)

#获取name对应的集合的所有成员

SCard (name)

#获取name对应的集合中的元素个数

R.scard ("Set_name")

Sinter (keys, *args)

# Get the set of multiple name corresponding sets

R.sadd ("Set_name", "AA", "BB")

R.sadd ("set_name1", "BB", "CC")

R.sadd ("set_name2", "BB", "CC", "DD")

Print (R.sinter ("Set_name", "set_name1", "set_name2"))

#输出: {BB}

Sismember (name, value)

#检查value是否是name对应的集合内的元素

Spop (name)

#从集合的右侧移除一个元素, and return it

Sunion (keys, *args)

#获取多个name对应的集合的并集

R.sunion ("Set_name", "set_name1", "set_name2")

Srem (name, value) removes an element from the collection

R.srem ("Set_name", "AA")

Hash type operation

Redis stores the hash type in memory as a dictionary stored in name.

Hset (Name,key,value)

#name对应的hash中设置一个键值对 (does not exist, is created; otherwise, modified)

#参数:

#name: Redis's name

#key: Key1 key in the corresponding hash

#value: value in the corresponding hash of the value1

#注: Hsetx (Name,key,value), created (equivalent to add) when the current key does not exist in the hash of name

Hget (Name,key)

#在name对应的hash中获取根据key获取value

Hmset (name,mapping)

#在name对应的hash中批量设置键值对

#参数:

#name: Redis's name

#mapping: Dictionary, as {' K1 ': ' v1 ', ' K2 ', ' v2 '}

#例:

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (R.hmget (' Test ', ' K1 ', ' K2 '))

#结果

[B ' v1 ', B ' v2 ']

Hmget (Name,keys,*args)

#在name对应的hash中获取多个key的值

#参数:

#name: Redis corresponds to name

#keys: To get a collection of keys, such as: [' K1 ', ' K2 ', ' K3 ']

#*args: The key to get, such as: K1,K2,K3

#例:

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (R.hmget (' Test ', ' K1 ', ' K2 ')) #获取多个值

Hgetall (name) : Gets the name of all the key values for the hash

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (R.hgetall (' Test '))

#结果:

{b ' k2 ': B ' v2 ', B ' K1 ': B ' v1 '}

Hlen (name) : Gets the number of key values in the hash for name

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (R.hlen (' Test '))

#结果:

2

Hkeys (name) : Gets the value of all keys in the hash corresponding to name

hvals (name) : Gets the value of all values in the hash of the name corresponding to

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (R.hkeys (' Test '))

Print (r.hvals (' Test '))

#结果:

[B ' K2 ', B ' K1 ']

[B ' v2 ', B ' v1 ']

hexists (Name,key) : Check if the hash name corresponds to the current key being passed in

Hdel (Name,*keys) : Deletes the key value pair for the specified key in the hash name corresponding to

#!/usr/bin/env python

#-*-Coding:utf-8-*-

Import Redis

Pool = Redis. ConnectionPool (host= ' 192.168.10.128 ', port=6379)

R = Redis. Redis (Connection_pool=pool)

R.hmset (' Test ', {' K1 ': ' v1 ', ' K2 ': ' V2 '})

Print (r.exists (' Test '))

Print (R.hdel (' Test ', ' K1 '))

Print (R.hmget (' Test ', ' K1 ', ' K2 '))

#结果:

True

1

Other common operations

Delete (*names)

#根据name删除redis中的任意数据类型

Exists (name)

#检测redis的name是否存在

Keys (pattern= ' * ')

#根据 *? Wait for wildcard matches to get the name of a Redis

Expire (name, time)

# Set a time-out for a name

Rename (src, DST)

# rename

Move (name, DB))

Move one of the Redis values to the specified db

# move one of the Redis values to the specified db

Type (name)

# Gets the type of the name corresponding value

20171124_python study six weeks five lessons (November 24)