My to python-day012-message middleware

Introduction to Message middleware The concept of message middlewareMessage middleware is the container that holds messages during message transmission. Message middleware acts as a middleman when relaying a message from its source to its target. The primary function of a queue is to provide routing and guarantee the delivery of messages, and if a message receiver is unavailable, Message Queuing retains the message until it can be successfully delivered, and of course, Message Queuing holds the message for a period of time. analogy to understand memoryUsing the relationship between producer and consumer to make analogy message producer → message middleware → message consumer ※Message is timeliness, there is a concept of expiration → message lease Message Middleware features 1, using asynchronous processing modeA message sender can send a message without waiting for a response.　　The message sender sends a message to a virtual channel (subject or queue), and the message recipient subscribes or listens for the channel. A message may eventually be forwarded to one or more message receivers, who do not have to respond synchronously to the sender of the message. The entire process is asynchronous. ※such as user information registration, after the registration is over a period of time to send mail or text messages. 2, application and application call relationships are loosely coupledSenders and recipients do not need to know each other, only to confirm that the message sender and recipient do not have to be online at the same time ※For example, online trading system in order to ensure that the final data consistency, after the payment system processing is completed, the payment results will be placed in the message middleware to notify the order system to modify the order payment status. The two systems are decoupled from the message middleware. Message Delivery service model transfer model of message middleware 1, point-to-point model (PTP)Point-to-point models are used for point-to-points communication between message producers and consumers. The message producer sends the message to a specific consumer identified by a name. This name actually corresponds to a queue in the message service, which is stored in the queue before the message is delivered to the consumer. Queue messages can be placed in memory or can be persistent to ensure that messages are still delivered when a message service fails.   Features:1, each message has only one consumer 2, the sender and receiver do not have time to rely on 3, the recipient acknowledges that the message received and processed successfully Model Diagram: 2, publish-subscribe model (PUB/SUB)The publisher/subscriber model supports the production of messages to a specific message topic. 0 or more Subscribers may be interested in receiving messages from a specific message topic. In this model, publishers and Subscribers do not know each other.This pattern is like an anonymous bulletin board .。 This pattern is summed up as: multiple consumers can get messages. There is a time dependency between the Publisher and the Subscriber. Publishers need to establish a subscription (subscription) to enable consumers to subscribe. Subscribers must maintain a continuous active state to receive messages unless the Subscriber establishes a persistent subscription. In this case (persistent subscription), messages published when the Subscriber is not connected will be republished when the Subscriber is reconnected. Features:1, each message can have more than 2 subscribers, publishers and Subscribers have time dependencies, the client only after the subscription to receive message 3, there is a durable subscription and non-durable subscription ① persistent subscription relationship is established, the message will not disappear, regardless of whether the Subscriber online ② non-durable subscription Subscribers must always be online in order to receive messages. Approximately equals point-to-point model when there is only one subscriber Model Diagram: case Study on the application scenario of Internet Message middlewareScenario 1: User registration asynchronous processing Case website user registration, after successful registration will send an email confirmation or SMS. Example: Scenario 2: Log analysis uses a case to collect the logs centrally for calculating PV, user behavior analysis.　　For example: Scenario 3: Data replication Case 1, copying data from source to multiple destinations, usually order or guarantee causality sequence. 2, users across the computer room data transmission, search, offline computing and so on. Such as: Scenario 4: Delay message sending and staging 1, the message middleware as a reliable message staging site 2, timed message delivery, such as the simulation of the user's second kill access, the system performance pressure measurement.Eg:tcpcopy,accesslog, etc.Example: Scenario 5: Message broadcast 1, cache data Synchronization Update 2, push data to the appeg: Update the local cache, for examplesuch as: Message Middleware Classification1. Push message type: Message producers send messages to the messaging service, and the messaging service pushes messages to consumers. 2. Pull message type: The consumer requests the message service to receive the message, and the message producer pulls the message from the message middleware. two different types of differences The essential difference between two types of push-and-pullwho saves the offset of the messagePush: The producer saves the push trajectory pull: the consumer saves the fetch state, the offset of the pulled position the message of middleware in combat MetaqIntroduction Metaq is a complete queue model message middleware, the server is written in the Java language, can be deployed on a variety of hardware and software platforms. The client supports the Java, C + + programming language. A single server can support more than 10,000 message queues, and by expanding the server, the number of queues can scale almost arbitrarily. Each queue is persistent, unlimited in length (depending on the size of the disk space), and can be consumed from any location in the queue. Official website: http://www.metaq.taobao.org/ Metaq Frame composition: Metaq features1, producers, servers and consumers can be distributed 2, message storage sequence write 3, high performance, throughput of 4, Support message Order 5,Client Pull, random read, bulk pull data 6, data migration, expansion to users transparent 7, consumption status saved on the client Metaq Important Terms 1, Topic: The subject of the message, defined by the user and configured on the server side. Producer send a message to a topic, consumer consume the message from a certain topic 2. Offset: Each partition of the message on the broker is organized into a list of files, and the consumer pulls the data to know the offset of the data in the file, which is called offset. Offset is Absolute Offset, the server converts offset to the relative offset of the specific file. 3. Broker: Meta servers, or server, are often referred to as brokers in message middleware. 4. partition (partition): The same topic is divided into several partitions, such as meta-test This topic we can be divided into 10 partitions, respectively, two servers are provided, then each server may provide 5 partitions, assuming that the server ID is 0 and 1, then all partitions are 0-0, 0-1, 0-2, 0-3 , 0-4, 1-0, 1-1, 1-2, 1-3, 1-4. Metaq Common Configuration ItemsZk.zkenable=true (whether registered to ZK, default to True) zk.zkconnect=localhost:2181 (ZK's server list) zk.zksessiontimeoutms=30000 (ZK heartbeat timeout, unit milliseconds,  Default 30 seconds) zk.zkconnectiontimeoutms=30000 (ZK connection timeout, unit milliseconds, default 30 seconds) Brokerid: (the server ID must be unique within the cluster and must be between integer 0-1024) ServerPort: (server port) HostName: (default will take the native IP (multi-machine NIC, need to specify)) Datalogpath: (Log data file path, the default is the same as DataPath) DataPath: (To specify the default data storage path) Del etepolicy=delete,168 (Data deletion policy, the default is more than 7 days to delete, here the 168 is the hour, 10s means 10 seconds, 10m means 10 minutes, 10h means 10 hours, the default is the hour) Deletewhen: (When to execute a cron expression that deletes a policy, the default is 0 0 6,18 *?, which is the daily morning and evening of 6 points to execute the processing policy.) Deletewhen: Delete policy execution time, cron expression) flushtxlogatcommit=1 (transaction log synchronization settings, 0 means let the operating system decide, 1 means each commit is synchronized, 2 means synchronization every 1 seconds, this parameter seriously affects transaction performance, A reasonable choice is made based on the tradeoff between performance and reliability you need. It is generally recommended to set to 2, which means to brush the disk every 1 seconds, that is, to lose at most one second of a run-time transaction. Such a reliable level is sufficient for most services. The safest of course is set to 1, but will severely affect transactional performance. and 0 has the lowest security level. The security level is 1>=2>0, while performance is 0 >= 2 > 1. Unflushthreshold: (Every number of messages do a disk sync, forcing the changed data to be brushed into the disk.) The default is 1000. This means that in the case of power-down, up to 1000 messages are allowed to be lost. can be set to 0, forcing each write to sync. When set to 0, the server is automatically enabled Group CommitTechnology that combines multiple messages into a single sync to improve IO performance. After testing, the TPS for the sender of the message was not affected too much by group commit, but the load on the server side increased a lot. Unflushinterval: (The default is 10 seconds for disk sync to be done periodically, in milliseconds.) That is, in the case of the server power-down, the maximum loss of 10 seconds sent over the message. cannot be set to less than or equal to 0) Metaq Installation Instructions1, first need to install configuration Zookeeper 2, install Java Runtime Environment 3, install Metaq Cluster Implementation method of MetaqStep1, all broker registers to zookeeper Step2, client connects to zookeeper, assigns a broker to customer the R abbitmq of the message middleware combat
RABBITMQ IntroductionRABBITMQ is a complete, reusable enterprise messaging system based on the AMQP protocol standard. He follows the Mozilla Public License open source agreement.    An industrial-grade Message Queuing (MQ) server implemented with Erlang. Official website: http://www.rabbitmq.com/ AMQP IntroductionAMQP (Advanced Message Queuing Protocol) is an application-layer protocol specification used by asynchronous messaging as a line-layer protocol rather than an API (for example, JMS) where AMQP clients can send and receive information regardless of the source of the message. The original purpose of AMQP is simply to provide the financial community with a message protocol that can collaborate with one another, and now the goal is to provide common building tools for the common Message Queuing architecture. Therefore, message-oriented middleware (MOM) systems, such as publish/subscribe queues, are not implemented as basic elements. Instead, by sending simplified AMQ entities, users are given the ability to build such entities. These entities are also part of the specification, forming a hierarchy at the top of the line layer protocol: the AMQP model. This model unifies the message pattern, such as the previously mentioned publish/subscribe, queue, transaction, and streaming data, and adds additional features such as easier to scale, content-based routing. RABBITMQ Overall Architecture

RABBITMQ Operating principle

RABBITMQ Important Terms1. Server (Broker): A process that accepts client connections to implement AMQP Message Queuing and routing capabilities. 2. Virtual Host: It is actually a dummy concept, similar to the rights control group, a virtual host can have several Exchange and queue, but the minimum granularity of permission control is virtual Host 3.Exchange: Accepts messages sent by the producer and routes the messages to the queues in the server according to the binding rules. ExchangeType determines the behavior of Exchange routed messages, for example, in RABBITMQ, ExchangeType has Direct, fanout and topicThree, the behavior of different types of exchange routes is not the same.　　4.Message queue: Message Queuing for storing messages that have not yet been consumed by consumers. 5.Message: Consists of header and body, header is a collection of various attributes added by the producer, including whether the message is persisted, which message queue is accepted, what priority is, and so on.　　The body is the app data that really needs to be transferred. 6.BindingKey: The so-called binding is the binding of a specific exchange and a specific queue, the binding keyword becomes bindingkey. Exchange Classification--Direct switch typeDirect Exchange – Handles routing keys directly and interactively. A queue needs to be bound to the switch, requiring the message to exactly match a specific routing key. This is a complete match. 　　　　If a queue is bound to the switch requiring a routing key (routing key) "Dog", only the message labeled "Dog" is forwarded, the Dog.puppy is not forwarded, the Dog.guard is not forwarded, and only the dog is forwarded. Graphic: Python code implementation (to be added) Exchange Classification--broadcast exchanger type fanout Exchange – Broadcast routing key. You simply have to bind the queue to the switch. A message sent to the switch is forwarded to all queues that are bound to the switch. Much like a subnet broadcast, each host in the network receives a copy of the message. The fanout switch forwarding message is the fastest illustration: Python code implementation (to be added) Exchange Classifications--topic-type exchanger Types Topic Exchange – A topic-based exchanger that routes messages to the bound queue through the pattern matching of the message's routing key and the binding keyword. This type of router can be used to support the classic publish/Subscribe messaging model--using the theme namespace as the message addressing mode to deliver messages to multiple consumers who partially or completely match the theme pattern. The topic Exchanger type works as follows: The binding keyword is made up of 0 or more tokens, with a "." Between each tag. Character separation. The binding keyword must be explicitly stated in this form, and wildcard characters are supported:" *" matches a phrase , "#" of 0 or more phrases . So the binding keyword "*.stock.#" matches the routing keyword "Usd.stock" and "eur.stock.db", but does not match the "Stock.nasdaq" plot: Python code implementation (to be added) RABBITMQ Installation

Step1, configuring Epel source

step2,yum-y Install Rabbitmq-server RABBITMQ Common ConfigurationIn general, the default configuration for RABBITMQ is sufficient. If you want special settings, there are two ways: one is the configuration file rabbitmq-env.conf of the environment variable, the other is the configuration file rabbitmq.config of configuration information; Note that these two files are not available by default and must be created yourself if necessary.　　1. rabbitmq-env.conf the location of this file is deterministic and immutable, located in:/etc/rabbitmq directory (this directory needs to be created by itself).　　Rabbitmq_node_ip_address: Specify IP address Rabbitmq_node_port: Specify the port number, default 5672 rabbitmq_config_file: The path to the configuration file, note that the profile suffix must be. CONFIG Abbitmq_log_base: Log file path 2, Rabbitmq.config This is a standard Erlang configuration file. It must conform to the standards of the Erlang profile. Erlang tuple, structure {Key,value}, key is the atom type, Value is a term, and several key parameters are: Tcp_listerners set RABBIMQ listening port, default is [5672]. Disk_free_limitDisk low watermark, which stops receiving data if the disk capacity falls below a specified value. Vm_memory_high_watermark, set the memory low watermark, if below the watermark, turn on the flow control mechanism, the default value is 0.4, that is, 40% of the total memory. Rabbitmq.config Configuration Example

[  {rabbit, [{    tcp_listeners,[{"127.0.0.1", 5672}]},    {ssl_listeners, [{"127.0.1.1", 5671}]},    {Ssl_ Options, [{CACertFile, "/usr/local/etc/rabbitmq/ssl/testca/cacert.pem"},                    {certfile, "/usr/local/etc/rabbitmq/ Ssl/server/cert.pem "},                    {keyfile,"/usr/local/etc/rabbitmq/ssl/server/key.pem "},                    {verify,verify_none},                    {fail_if_no_peer_cert,false}]}]  }].

RABBITMQ Command Introduction1,/etc/init.d/rabbitmq-server Start | Stop | Restart | Reload 2, Rabbitmqctl add_vhost vhostname create Vhost:3, rabbitmqctl delete_vhost vhostname Delete vhost 4, RABBITMQCT L list_vhosts Traverse all virtual host Information 5, RABBITMQCTL add_user username password Add user and password 6, Rabbitmqctl change_password username n　　Ewpassword Modify User Password 7, rabbitmqctl set_permissions-p v_host user ". *" ". *" ". *"//bind permission, and have read and write permissions. 8. Rabbitmqctl list_queues//Show All queues

My to python-day012-message middleware