Design of subscription Publishing system based on HTTP protocol

 design of subscription publishing system based on HTTP protocol --Architecture design of Internet of things system   1, Subscription release (Subscriber-publisher)The most typical application scenario for a subscription publishing model is the design of a messaging system. In the schema of the message system, the sender of the message is called (publisher), and the recipient of the message is called (subscriber), see Wikipedia:publish–subscribe pattern. The architecture of the entire messaging system can be described as:

Figure 1 shows that the message system consists of 3 components: the publisher, the Subscriber, and the message broker (broker), while the core of the entire message system is the broker, and the difficulty of the broker's implementation in terms of business capability is mainly its throughput. Take the mobile phone message push example, in the current era of mobile interconnection, most of our very common app users are basically millions (popular app is tens), which means that the broker should be able to support at least the subscription of millions of devices, the use of a single server broker obviously does not solve the problem. In the era of the Internet of Things, subscribers will no longer be mobile only, subscribers can be any electronic device, and the level of this scenario will be a hundredfold of the number of mobile phones. the implementation scheme of 2,MQTT Protocol's publish and subscribe system 2.1,MQTT ProtocolAccording to the official definition, the MQTT protocol is machine-to-machine Connection protocol, which is a very lightweight message Transfer Protocol designed for the publish subscription model. For details, see:http://mqtt.org/from the MQTT protocol definition, this MQTT is designed for publishing a subscription system and is very lightweight. 2.2, the implementation of the schemeImplement a complete set of release subscription systems, primarily two components (client and broker), one protocol specification (MQTT). The current popular open source MQTT client implementation is PAHO (HTTP://ECLIPSE.ORG/PAHO); Popular open source MQTT broker implementations include Apache Apollo and Eclipse Mosquitto (Http://eclipse.org/mosquitto), Mosquitto's advantage is very lightweight, using a Raspberry Pi (or router) Such a small device is enough to serve a home device connection. 2.3, Architecture designThe service system architecture for publishing subscriptions is simple and basically follows the basic architectural pattern of Figure 1. For a home IoT application, if the device only wants access within the LAN, the broker only needs to be installed on a small device (Nanopi or RASPI-based) or directly integrated into the router. Of course, for real IoT applications, we still want devices to be managed and controlled over the internet, so many brokers should actually be in an Internet server. 2.4, the subscription publishing system interaction principle of the MQTT protocolFirst quote the Open Source project PAHO provides a Python version of the client to perform subscription and release action demo, the code is very short

1 #Susbscriber2 ImportPaho.mqtt.client as Mqtt3 4 #The callback for when the client receives a CONNACK response from the server.5 defon_connect (client, UserData, RC):6Client.subscribe ("$SYS/#")7 8 #The callback a PUBLISH message is received from the server.9 defon_message (client, UserData, msg):Ten     Print(msg.topic+" "+str (msg.payload)) One  AClient =Mqtt. Client () -Client.on_connect =On_connect -Client.on_message =On_message theClient.connect ("iot.eclipse.org", 1883, 60) -   - #Blocking call that processes network traffic -Client.loop_forever ()

View Code

Subscriber: The Subscriber client code shows that subscribers only need to do 2 actions (connecting broker and establishing a long loop of waiting) and providing 2 interface functions (subscription request functions and functions that process the results of the broker response). The basic features simply request a connection, subscribe to the specified topic message, and process the response result, but Loop_forever () is an infinite loop, which means that the client and Borker maintain a long socket connection. So from here you can realize that one of the bottlenecks of broker is how many such long connections can be handled.

1 #Publisher2 ImportPaho.mqtt.client as Mqtt3  4Client =Mqtt. Client ()5Client.connect ("iot.eclipse.org")6 Client.loop_start ()7res = Mqttc.publish ("$SYS/#","HELLO")8Client. Loop_stop (Force=false)

Publisher: Publisher client code shows that Publisher actions are simpler than subscribers, and the basic elements are nothing more than establishing a connection, publishing a specified topic message to the broker, and ignoring the result response process. The nature of the interaction logic of Subscriber and publisher is based on the TCP protocol socket implementation, opening the MQTT protocol port for the server socket, and opening an asynchronous thread to continuously listen on the port, waiting for the client side ( Subscriber and publisher) the socket for the MQTT request, the client side of the Subscriber MQTT request is somewhat different, that is subscriber socket actual and the server has maintained long connection, Always wait for the server to push over the message until the connection is closed. So aside from the details of the problem, you can use the Netty framework, based on the MQTT protocol quickly developed a set of server and client-side applications. 3,http Protocol broker design Implementation

Figure 2 Subscription release System Broker design   HTTP protocol and MQTT protocol comparison:          Pros: HTTP is the most widely used in the Internet era, fully testing its effectiveness and stability, Full community support and mature open source resources available           bugs: The relative MQTT protocol is too heavy for network requirements, and directly based on HTTP is not able to implement publish subscriptions (HTTP is a simplex protocol that relies on WebSocket, servlet3.0 and other technologies to achieve duplex)       This is the implementation of the HTTP protocol-based publish/subscribe system broker using servlet3.0 technology, as shown in Figure 2 for the IoT broker system design architecture. The backend broker is divided into two modules: the Release Center (user and device) and the subscription Center (user and device), and the event bus. Such a design may be confusing, why not directly abstract into the event's release and subscription center, so soon with the Mqtt broker agree? Indeed, since it is implemented using the HTTP protocol, why is it completely modeled on the MQTT protocol, and what we are designing is actually a "IoT business system" rather than a "middleware", so if you change a business scenario, you have to redesign the system, And it happens that the application of Servlet based on HTTP protocol is a rich open source resource for the business system.    below explains in detail the design of the user Publishing center and the subscription center, because in the IoT scenario, the primary business interaction logic is to do publish and subscribe.  User Publishing Center (publisher) between users and devices:       In the IoT scenario, the user acts as the core business publisher, for the broker's release center, The data that is received by all front-end user requests will be encapsulated as an event broadcast to the subscription center by the publishing center in the broker's internal system. For example, the app is Publisher's terminal, and the core business logic of the motorcycle is the lock command and a series of trading logic. In terms of unlocking action, the release center received the unlock event, before publish this event, different business scenarios for this event may have different business requirements, the typical scenario is whether the event requires mass, whether the event requires a timing function, whether the event needs to be published reliably. In particular, the reliable release of events is an essential requirement in trading class systems. In the case of a motorcycle, the lock order will start to takeWhen billing and withholding money, this time it is necessary to rely on broker in the application layer to face data to do business assurance, and not rely on the stability of basic system services.   Subscription Center (subscriber):      Design for a subscription center (either a user or a device) that is designed in full accordance with the data structure of table or key-value, i.e., for each request, The broker will associate a handler with its corresponding ID. When the event is published to the subscription center, the processor of the subscription center uses the event ID (or uniquely identified device ID) to query the corresponding handler and respond to the results. Because it is based on the HTTP protocol, you need to rely on servlet3.0 or websoket technology for implementation.   4, Domain modeling 4.1 Release Center domain modelingThe core feature in publishing is publishing events, so event is the core domain object of the release center. As already stated in Figure 2, the basic functional elements required for event publishing, the event design is mainly to achieve the functions described in part 3rd. Figure 3 Release Center domain model abstraction in Figure 3, Abstractevent is the entity abstraction design at the top of the event. Because the publishing center may publish many different types of events, abstractevent must have the EventType attribute to describe the type of event. Whether that type of event is actually an entity, since it is an entity that has its own id,eventid as the unique identifier of the event, it needs to have a clear explanation that the meaning of EventID is actually equivalent to topic, This means that not every event that is published generates a new EventID. For example, in the application of the motorcycle, the lock is a kind of event, for each bike, there is a unique EventID. Abstractevent defines the cronexpression and Deliverystatus attributes for events that are mentioned earlier in the 3rd section, where the event requires a cycle, delay, and reliable release, where cron expressions can be very concise to describe and implement event settings for periods and delays , and Deliverystatus needs to use state to guarantee the transaction of event action in distributed network environment. Additionally, the definition of groupevent is intended to address the issue of publishing an event that is mentioned in part 3rd in response to multiple devices. 4.2 Subscription Center domain modelingThe core abstraction of the subscription center domain is handler, where each handler corresponds to a subscription HTTP request. Each subscription request Handler holds the EventID that it wants to respond to, the business data it carries, and the callback method that responds to the result. Subscribers expect that when an event of the EventID identity occurs, the corresponding incident response can be received immediately, or that the subscription HTTP request is a network connection that remains waiting for a long time. So all handler should have a holder to cache it for management, which is the meaning of the handlerholder existence of singleton patterns. For Handlerholder, there are two options for the handler cache policy: 1, cached in table Form, 2, cached in map form. Compared to the two cache strategies have advantages and disadvantages, table Form saving storage but find expensive (can be ordered to improve speed), map form lookup fast but consume storage, but regardless of that form cache can be through the hierarchical cache to improve the cache capacity (such as primary cache summary data in the system memory,  Secondary cache primary data in the Redis cache system). The status diagram in Figure 4 depicts the event-driven handler transfer process (user and device) in the subscription center. At the initial moment, the device initiates a subscription to the CommandEvent request, waiting for the release center to receive a commandevent request from the user, at which point the release center will determine if the event delivery status needs to be logged, such as the need for a device response okevent, The corresponding handler will be generated to the subscription center. At this point, the handler that responds to the device will carry the deliverystatus=waiting identity, waiting for the device to return the confirmation result. The acknowledgment response returned by the device can then be published okevent in response to user processing results through the release center. (The actual processing process should be more complex, because the exception is not considered, such as the device does not receive the response results, the preparation response results are lost, and so on, these need to do some compensation strategy)

Figure 4 Subscription Center Domain modeling Abstraction

Design of subscription Publishing system based on HTTP protocol