Beyond polling? Consider pubsub, push, and Mom

This article Reprinted from: http://www.infoq.com/cn/news/2008/08/beyondpolling

 

InOscon' 08At the conference,Evan 'rabble 'Henshaw-PlathAndKellan Elliott-McCreaIntroduced 《Beyond rest? Use XMPP pubsub to build data services.Robert KayeThis slide is reported as follows:

Kellan talked about friendfeed , it is a website that allows users to know that their friends share the new project. In this example, Kellan points out that in order to check for updates for 45000 users, friendfeed goes to Flickr 2.9 million polling times. Of the 45000 users, only 6700 are logged on at any time. This is of course a poor method for content update checks. Kellan said: "polling is too bad !"

the key to solving this problem is to leave the standard rest web service behind, find a way to use message transmission, which is a direct communication method that notifies users of content changes.

In this context, polling means usingA restful Web ServiceFor each userGet)Update. In contrast,Pubsub(Publish/subscribe, publish/subscribe) is a method that uses the asynchronous message passing protocol. The publisher and any subscriber are decoupled. When an update notification needs to be sent to a large number of clients, these features make pubsub an appropriate scalability option.

In this slide, Evan and rabble describedJabber(Based onXMPP[Extensible messaging and presence protocol, scalable messaging and on-site protocols] pubsub Service) Advantages:

1. XMPP works on persistent connections
2. It is stateful (SSL becomes cheaper)
3. designed as an event stream protocol
4. Federal and asynchronous nature
5. built-in identity authentication, security, and presence
6. A Jabber server is built and deployed for this purpose.

This slide has caused great repercussions. Kirk Wylie considers image amqp This is based on mom (mesage oriented middleware, message-oriented middleware) and Joshua Schacter (Del. icio. founder of US) also joined the debate and pointed out a simpler method. This method uses the HTTP callback-based technology:

In short, unlike frequent polling, a client can send a common HTTP request containing the endpoint for resource subscription (resouce) and delivery update:Http://your.app/subscribe? Resource =/Some/user & callback = http://my.app/endpoint

In this way, the endpoint receives the RSS entry fragment only when the resource is updated. For the sake of security, the exchange must contain a token, which can be used for reference by appropriate protocols. Subscription will expire after a period of time, for example, 24 hours, or pass it as a parameter.

Comments noted thatSuch a system already exists:Webhooks.

Rabble provides hisThoughts:

There are two problems here. We have considered adding a slide to discuss the ping back system. Apparently, the ping back/Web hook operation seems to be a good mode, but it may also be an anti-pattern?

It is foreseeable that you will face aggregator/crawler problems when creating a large-scale Web hook system. XMPP is absolutely feasible. In this way, we not only implement federated functions, but also have better interfaces. We also have the potential to add some proxy authentication (auth) functions.

Methods such as webhooks are potentially much easier than using Jabber/XMPP,According to Blane cookThis complexity makes sense:

If we claim that we only need 10 lines of PHP scripts to make the system available, we can only get one Output Message Queue with poor implementation. In the case of Medium Scale (10000 users, each with 50 contacts, 1/5 offline, and 2 posts per day), you will see 2.3 remote HTTP requests per second, this method is not useless.

Although using pubsub for notification is an excellent architectural method, many people have doubts about the title of the slides.Dare Obasanjo summarized this very well., Indicating that rest is not a"Golden HammerA hammer in your hand, and everything you see is a nail .) :

[Therefore] This example does not mean that rest cannot be scaled as Evan and Kellan's lecture titles imply, but an example of using an error tool to solve the problem. The reason for choosing it is that it happens to work well on other occasions.

Even if there is no other content, this discussion also highlights this fact, considering that the use model of API consumers plays a heavy lifting role in determining the proper design.