Introduction to Service Broker

Since cross-domain file transmission is recently underway, SQL broker is used for a brief introduction, most of which are from msdn.

Service broker is a new technology in Microsoft SQL Server 2005 that helps database developers generate secure, reliable, and scalable applications. As service brokers are an integral part of the database engine, managing these applications becomes part of routine database management.

Service Broker provides a queue and reliable message transmission for SQL Server. Service broker is applicable to applications that use a single SQL server instance and applications that are assigned to multiple instances.

In a single SQL server instance, Service Broker provides a reliable asynchronous programming model. Database applications usually use asynchronous programming to shorten the interactive response time and increase the total throughput of applications.

The service broker also provides reliable message transmission services between SQL Server instances. Service Broker helps developers write applications related to independent and self-contained components called services. Applications that need to use the functions of these services can use messages to interact with these services. The Service Broker uses TCP/IP to exchange messages between instances. Service Broker provides some functions to prevent unauthorized access over the network and encrypt messages sent over the network.

1. The role of Service Broker: Service Broker helps developers generate asynchronous loosely coupled applications that can combine independent components to complete tasks. These application components exchange messages that contain the information required to complete the task.
2. Session
3. Message sorting and coordination
4. Asynchronous transaction Programming
5. Supports loosely coupled applications
6. Service Broker component
   • Service Broker queues are integrated into databases.
   • The queue coordinates and sorts related messages.
   • Session component. Session groups, sessions, and messages constitute the runtime structure of the Service Broker application. The application exchanges messages as part of a session. Each session is part of a session group. A session group can contain multiple sessions. A service broker session is a dialog, that is, a session between two participants. For more information about session components, see session architecture.
   • Service definition component. These components are design-time components used to specify the basic structure of the session used by the application. They define the Message Type of the application, the session stream of the application, and the database storage of the application. For more information about service definition components, see service architecture.
   • Network and security components. These components Define the infrastructure for exchanging messages outside the SQL server instance. To help the database administrator manage and change the environment, the Service Broker allows the Administrator to configure these components independently of application code. For more information about network and security components, see network transmission and remote security.

Session:

Service broker is designed around the basic functions of sending and receiving messages. Each message is an integral part of a "session". A session is a reliable and persistent communication channel. Service Broker requires that each message and session must have a specific type to help developers write reliable applications.

The new Transact-SQL statements allow applications to send and receive messages reliably. The application sends a message to the service. The service is the name of a group of related tasks. An application receives messages from a "queue", which is a view of an internal table.

Messages of the same task are part of the same session. In each session, the Service Broker ensures that the application only receives each message once and receives the message according to the sending order. Programs that implement a service can be associated with sessions of the same service in the "session Group". For more information, see advantages of service broker.

The certificate-based security mechanism helps protect sensitive messages and control service access.

One way to understand service broker is to think of it as a postal service. If you want to start a conversation with a remote colleague, you can send a mail via the postal service, which sorts and delivers the mail. You and your colleagues retrieve, read, write, and send new emails from your mailbox until the session ends. Mail transmission is "Asynchronous", that is, you and your colleagues can process other tasks at the same time.

In this analogy with the postal service, a letter is a message. The Service Broker service is the address where the post office delivers the mail. A queue is the email address used to save mails after they are delivered. The application receives messages, operates on messages, and sends responses.

The program using service broker maintains sessions with other programs in a similar way as the mail.

You do not need to know when your colleagues will read or reply to the email. Similarly, applications using service broker do not have to understand how other services process messages, how messages are transmitted, or when other applications process messages.

Message sorting and coordination:

When processing queues (a common database programming technology), service brokers have two key aspects different from traditional products:

 

The integration queue means that regular database maintenance and management also includes service broker. Generally, administrators do not need to perform routine maintenance tasks related to service brokers.

The Service Broker framework provides a simple Transact-SQL interface for sending and receiving messages, which combines a set of strong assurance policies for message transmission and processing. The Service Broker ensures that the program only receives each message in the session once. The order of message sending is not the order in which the message enters the queue. The traditional queue product provides the order in which messages enter the queue, and requires the application to determine the order and group of messages. Service Broker ensures that the two queue readers cannot simultaneously process messages from the same session or related session groups.

The START program starts a session for each task and sends a message to the target service. The message contains the data required to perform specific steps in the task. The target service receives the message. The program of the target service processes the message and then responds to the starting service. The session continues and ends based on the rules determined by the developer.

The service broker can process the most difficult tasks included in the compilation of message applications. These tasks include coordinating, reliable message delivery, locking, and starting the Queue Reader. This allows database developers to focus on solving business problems.

Asynchronous transaction Programming

In the basic structure of service broker, the message transmission between applications is "transaction" and "Asynchronous. Because the Message Passing of service broker is transactional, if a transaction is rolled back, all service broker operations in the transaction will be rolled back. This includes sending and receiving operations. In asynchronous transmission, the database engine processes and transmits data as the application continues to run. To improve scalability, Service Broker provides some mechanisms that can automatically start these programs when the program processing the queue needs to do some necessary work. For more information, see service broker activation.

Asynchronous programming helps developers write applications that use queues. Many databases contain tables used as queues, including the work to be done when resources permit. The queue allows the database to maintain a response to the interactive user when effectively using available resources. Service Broker provides the queue function as an important part of the database engine.

The queue function allows an application to execute this job in another transaction that is different from the transaction in which the request is made. Service broker extends this concept so that applications can perform work on another instance or on another computer. Service brokers provide reliable transaction message transmission between built-in queues and instances in the database, which is very helpful to database developers.

Supports loosely coupled applications

Service broker supports loosely coupled applications. Loosely Coupled applications are composed of multiple programs that independently send and receive messages. Such applications must contain the same definitions used to exchange messages, and must define the same overall structure for interactions between services. However, applications do not have to run at the same time, run in the same instance of SQL Server, or share implementation details. The application does not have to know the physical location or implementation of other participants in the session.

Service Broker component

Service Broker has three types of components:

 

Service Definition components, network components, and security components are part of the metadata of databases and SQL Server instances. Session groups, sessions, and messages are part of the data contained in the database.

2. The service broker function is of great help to database applications. These features and benefits include:

• Database Integration improves application performance and simplifies management.
• Message sorting and coordination for simplified application development.
• Application loose coupling provides workload flexibility.
• Message locks enable multiple instances of an application to synchronously process messages in the same queue.
• Automatic activation allows the application to adjust with the amount of messages.

The service broker function is of great help to database applications. These features and benefits include:

 

Database Integration

The integration design of service broker is helpful for application performance and management.

Integration with SQL server enables transactional message transmission, which avoids extra overhead and complexity of the external Distributed Transaction Coordinator. In a database transaction, an application receives one or more messages, processes the messages, and sends a reply message. If the transaction fails, all work is rolled back, And the received message is returned to the queue, so that you can try to process it again. The operation takes effect only after the application commits a transaction. The application remains consistent.

When data, messages, and application logic are in the database, management becomes simpler, because Application Management (disaster recovery, security, backup, and so on) becomes a part of routine database management, the Administrator does not have to manage three or four independent components.

In traditional message passing systems, the message storage zone and database can be different. For example, when a component is restored from the backup, The other component must also be restored from the backup. Otherwise, the information in the message storage area does not match the information in the database. Because the service broker saves messages and data in the same database, inconsistency is eliminated.

A public development environment is also a benefit of database integration. The messaging and data parts of an application can use the same SQL Server language and tool in the service broker application, this allows developers to fully utilize the knowledge of the Database Programming Technology in message-based programming. The stored procedure for implementing the Service Broker service can be written in transact-SQL or a Common Language Runtime (CLR. Programs outside the database use Transact-SQL and similar database programming interfaces, such as ADO. net.

In addition, database integration makes automatic resource management possible. The Service Broker runs in the context of the SQL server instance, so it monitors all messages to be sent from all databases in the instance. In this way, each database can help the service broker manage the resource usage of the entire SQL server instance while maintaining its own queue.

Sort and coordinate messages

In traditional message passing systems, applications are responsible for sorting and coordinating messages that may not arrive in the order. For example, application a sends messages 1, 2, and 3. Application B receives and confirms messages 1 and 3, but an error occurs when receiving message 2. Application A resends Message 2, but now the message is received after Message 1 and 3. In the past, developers may have to write applications so that there is no problem with the order of messages, or temporarily cache Message 3 before Message 2 arrives, so that the application can process these messages in the correct order. Both solutions are neither clear nor simple.

The traditional system still has another problem, namely repeated transmission. In the preceding example, if application B receives Message 2 but the confirmation message returned to application a is lost, application a resends Message 2, this causes application B to receive the message twice. The application code must be able to identify a duplicate message and discard it, or re-process the duplicate message without any negative effect. Likewise, these two methods are hard to implement.

Message coordination is traditionally hard to handle. For example, an application may submit hundreds of thousands of requests to a service. The service processes these requests in parallel and returns a response immediately after each request is processed. Because the duration of processing each request is different, the order in which the application receives the response is different from the order in which it sends the outgoing message. However, to correctly process the response, the application must associate each response with the correct outgoing message. In traditional message passing systems, each application manages this association, which increases the development cost and complexity of applications.

The Service Broker solves these problems by Automatically Processing the message sequence, unique transmission, and session ID. After a session is established between two service brokers, an application receives each message only once and receives the message in the sending order. You do not need to write additional code for the application to process these messages in order and only process them once. Finally, the Service Broker automatically includes the identifier in each message. The application can always know which session the specific message belongs.

Loose coupling and workload flexibility

Service Broker provides loose coupling between the starting application and the target application. An application can send a message to the queue and then process it. Instead, the Service Broker ensures that the message reaches its target. This loose coupling brings scheduler flexibility. The initiator can send multiple messages, and multiple target services can process these messages in parallel. Each target service processes these messages at its own speed, depending on the current workload.

Queuing also allows the system to evenly allocate processing tasks, reducing the peak capacity required by the server. This improves the overall throughput and performance of database applications. For example, many database applications increase the volume of transactions in certain time periods of a day, which increases resource consumption and shortens response time. With service broker, these applications do not need to perform all the processing of the transaction for the committed business transactions. Instead, the Service Broker sends information about the transaction to the application that executes background processing. The applications that execute background processing reliably process these transactions for a period of time, while the main portal application continues to receive new business transactions.

If the target is not immediately available, the message is retained in the transmission queue of the sending database. The Service Broker retries to send the message until the message is successfully sent, or until the session lifetime expires, even if one of the two services is unavailable at a certain point during the session, the session can also be reliably continued between two services. Messages in the transmission queue are part of the database. Even if the instance fails over or is restarted, the service broker can transmit messages.

Message lock

In traditional message passing applications, one of the most difficult tasks is to allow multiple programs to read the same queue in parallel. In traditional message passing systems, when multiple programs or threads read the same queue, messages may be processed in wrong order. Service Broker uses session Group locking to prevent this situation.

Imagine the traditional order processing application. The queue receives two messages. Message A contains instructions for creating order titles, and message B contains instructions for creating order lines. If the two messages are removed from the queue by different application instances and processed at the same time, the order line transaction may attempt to submit first and fail because the order does not exist. This failure leads to transaction rollback and re-queue of messages, and the messages are processed again, wasting resources. The traditional approach for programmers to solve this problem is to combine the information groups in message A and message B into a message. Although this method is simple and clear for two messages, it is powerless for systems that need to coordinate dozens or hundreds of messages.

The Service Broker solves this problem by associating sessions in the session group. Service Broker automatically locks all messages in the same session group so that these messages can only be received and processed by one application instance. At the same time, other instances of the application can continue to remove messages from other session groups from the queue and process them. This allows multiple parallel application instances to work reliably and effectively without the need to write complex lock code in the application.

Automatic activation

One of the most useful functions of service broker is activation. The activation function allows the application to dynamically adjust itself to match the number of messages received in the queue. Service Broker provides some functions that enable activation of programs running in the database and those running outside the database. However, service brokers do not require activation for applications.

The Service Broker monitors activities in the queue to determine whether an application is receiving messages for all sessions containing available messages. When a job needs to be executed by a new Queue Reader, the Service Broker activation function starts a new Queue Reader. The Service Broker monitors the activities in the queue to determine when a job needs to be executed by the Queue Reader. When the number of queue readers matches the incoming communication traffic, the queue periodically enters a status, that is, the queue is empty or, or all messages in the queue belong to sessions that are being processed by another Queue Reader. If the queue does not reach this status within a period of time, the Service Broker activates another instance of the application.

For programs running in the database and those running outside the database, the application uses different activation methods. For programs running in the database, the Service Broker starts another copy of the stored procedure specified by the queue. For programs running outside the database, the Service Broker generates an activation event. The program monitors this event to determine when another Queue Reader is required.

The service broker does not stop programs started through the activation function. All the activated applications are written as if no messages arrive for receiving within a specific period of time, they will be automatically closed after this period of time. Applications designed in this way allow the number of application instances to dynamically increase or decrease as the communication traffic of the service changes. In addition, if the system is disabled or restarted, when the system restarts, the application automatically starts to read messages in the queue.

The traditional message passing system does not have this behavior. The results are usually too many or too few resources dedicated to a specific queue at a given time.

3. Typical use of Service Broker

• Asynchronous triggers
• Reliable Query Processing
• Reliable data collection
• Distributed Server processing for client applications
• Used for data merging of client applications
• Large-scale Batch Processing

Service broker is useful for applications that require asynchronous processing or applications that need to be distributed across multiple computers. Typical use of service broker includes:

 

Asynchronous triggers

Many applications that use triggers (such as online transaction processing (OLTP) Systems) can benefit from service brokers. The trigger queues messages requested by the service broker. The trigger does not actually execute the requested work. A trigger creates a message that contains information about the job to be completed and sends the message to the service that executes the job. The trigger then returns.

When the original transaction is committed, the Service Broker passes the message to the target service. The program implementing this service executes this task in a separate transaction. By performing this operation in a separate transaction, the original transaction can be committed immediately. Applications can avoid system speed degradation caused by keeping original transactions open when performing this operation.

Reliable Query Processing

Some applications must be able to handle queries reliably in the case of computer failures, power outages, or similar problems. Applications that require reliable query processing can submit queries by sending messages to the Service Broker service. The application implementing this service reads the message, runs the query, and returns the result. The preceding three operations occur in the same transaction. If a fault occurs before the transaction is committed, the entire transaction will be rolled back and the message will be returned to the queue. When the computer recovers, the application restarts and processes the message again.

Reliable data collection

Applications that collect data from a large number of sources can use service brokers to reliably collect data. For example, retail applications with multiple sites can use service brokers to send transaction information to the central data storage area. Because the Service Broker provides reliable asynchronous message transmission, you can continue to process transactions even if the sites are temporarily disconnected from the central data storage area. The Service Broker security mechanism can help ensure that messages are not mistakenly transmitted, and help protect data in transmission.

Distributed Server processing for client applications

Large applications that Access Multiple SQL Server databases can benefit from service brokers. For example, a Web application that subscribes to a book can use the server-side service broker to exchange information between different databases including ordering, customer, inventory, and credit card data. Because service brokers provide message queues and reliable message delivery, applications can continue to accept orders even if a database is unavailable or overloaded. In this case, the service broker is used as the framework of the distributed OLTP system.

Used for data merging of client applications

Applications that must use or display information in multiple databases can use service brokers. For example, customer service applications that merge data from multiple locations to one screen can use service brokers to run these requests in parallel (rather than sequentially, this can greatly shorten the application response time. Customer service applications send requests to different services in parallel. When these services respond to requests, customer service applications collect responses and display results.

Large-scale Batch Processing

Applications that must execute large-scale batch processing can use the queuing and parallel processing functions provided by the Service Broker to quickly and efficiently process a large amount of work. The application stores the data to be processed in the service broker queue. The program regularly reads and processes data from the queue. Applications can use reliable message transmission provided by the Service Broker to execute batch processing on computers other than the computers sending requests.