Lightweight open source simple Queue service based on HTTP protocol: httpsqs[to]

The

Httpsqs (HTTP simple queue service) is a lightweight, open-source, easy-to-use Message Queuing service based on the HTTP Get/post protocol that uses the Tokyo Cabinet b+tree key/value database for data persistence Storage.

Project URL: http://code.google.com/p/httpsqs/
Working with documents: http://blog.zyan.cc/httpsqs/
Usage Environment: Linux (supports 32-bit, 64-bit operating system, 64-bit operating system is recommended)
Software feast

Queues (queue) are also known as first-in-one-out tables, which first enter the elements of the queue, first taken out of the queue. The part of the element is called "Team Head" and the end of the element is called "Team Tail". Message Queuing is a good way to handle data transfer and storage asynchronously, and when you frequently insert data into a database and submit data to a search engine frequently, you can take Message Queuing to insert asynchronously. In addition, the slow processing logic, the processing logic with concurrent quantity limit, can be processed in the background through message queue, such as FLV video conversion, sending SMS, sending e-mail, etc. The

HTTPSQS has the following characteristics:

is very simple, based on the HTTP Get/post protocol. PHP, Java, Perl, Shell, Python, Ruby and other programming languages that support the HTTP protocol can be called. The
is very fast, in-queue, out-of-queue faster than 10,000 times/second.
High concurrency, support tens of thousands of concurrent connections, c10k is not a problem. The
supports multiple queues.
The maximum number of queues supported by a single queue is up to 1 billion.
Low memory consumption, massive data storage, storage of dozens of GB of data in less than 100MB of physical memory buffer. The
can easily modify the maximum number of queues for a single queue without stopping the service. The
can view the status of the queue in real time (in-queue, out-of-queue, unread-queue, maximum-queue count). The
can view the contents of the specified queue ID (queue point), including the contents of the queue that are not out and out. The
supports multi-character-set encoding when viewing queue content. The
source code does not exceed 800 lines and is suitable for two development.

1, httpsqs 1.7 pressure test:

Use the Apache AB command for stress testing, open 10 threads, put 100,000 text data (each 512 bytes) into the queue:
using HTTP Keep-alive: 23018 requests/sec
Turn off HTTP keep-alive: 11840 requests/sec

Stress test with Apache AB command, open 10 threads,　　Remove 100,000 Text data (512 bytes per piece) from the queue:
When using HTTP keep-alive: 25982 requests/sec
When you turn off HTTP keep-alive: 13294 requests/sec

Detailed test content: Http://code.google.com/p/httpsqs/wiki/BenchmarkTest

Production Environment application: in Jinshan Game official website, news, forum posts, customer service announcements, SNS community, such as the occurrence of increased, deleted, Change operation, the text content writes HTTPSQS queue in real time, the whole station search engine incremental index quasi real-time (within 1 minutes) updated data source is taken from HTTPSQS. Httpsqs December 18, 2009 up to date, the operation is stable, both inbound and outbound from the Web server, as well as batch-in and out-queue operations from the command line script.

　　2, HTTPSQS production environment application:

Jinshan Pass (https://my.xoyo.com)
Queue Application Type: Mobile SMS uplink, mobile phone message issued, Mail issued
Queue application Requirements: high stability, large amount of storage data
Queue deployment structure: One master, one standby two HTTPSQS hot standby mode

Kingsoft User Behavior Analysis System (http://kbi.xoyo.com)
Queue Application Type: User mouse click, access URL raw data collection
Queue application Requirements: High concurrency performance, large storage data volume
Queue deployment Structure: Multiple HTTPSQS application layer hash distributed mode

Jinshan Network game operating platform Kingeyes
Queue Application Type: User Action logging

Search in Jinshan Getaway website
Queue Application Type: Index quasi-real-time update. In the official website of Jinshan Games, news, forum posts, customer service announcements, SNS community, such as the increase, deletion, change the operation, the text content in real-time write Httpsqs queue, the whole station search engine incremental index quasi-real-time (within 1 minutes) updated data source from Httpsqs.

The general comment system of Jinshan Xiaoyao net whole station
Queue Application Type: Comment Post

Jinshan "Swordsman Love" TV series four characters talent draft activity (http://zt.xoyo.com/haixuan/)
Queue Application Type: User-uploaded photos asynchronous cropping, scaling processing

Sina Email (http://mail.sina.com.cn)
Queue Application Type: User login log record

　　3. Httpsqs Compile and install:

Ulimit-shn 65535wget Http://httpsqs.googlecode.com/files/libevent-2.0.12-stable.tar.gztar ZXVF LIBEVENT-2.0.12-STABLE.TAR.GZCD libevent-2.0.12-stable/./configure--prefix=/usr/local/libevent-2.0.12-stable/ Makemake INSTALLCD. /wget Http://httpsqs.googlecode.com/files/tokyocabinet-1.4.47.tar.gztar ZXVF TOKYOCABINET-1.4.47.TAR.GZCD Tokyocabinet-1.4.47/./configure--prefix=/usr/local/tokyocabinet-1.4.47/#注: Compile the Tokyo cabinet on a 32-bit Linux operating system, using the./ Configure--ENABLE-OFF64 instead of./configure, you can make the database file exceed the 2GB limit. #./configure--enable-off64--prefix=/usr/local/tokyocabinet-1.4.47/makemake INSTALLCD. /wget Http://httpsqs.googlecode.com/files/httpsqs-1.7.tar.gztar zxvf HTTPSQS-1.7.TAR.GZCD httpsqs-1.7/makemake Installcd. /

　　

　　4. HTTPSQS Server usage Documentation:

　　

[Email protected] ~]# httpsqs-h

-L <ip_addr> Listen IP address, default value is 0.0.0.0
-P <num> Listening TCP port (default: 1218)
-X <path> database directory, directory does not exist will be created automatically (for example:/opt/httpsqs/data)
-T <second> http request time-out (default: 3)
-s <second> number of seconds between synchronizing memory buffer contents to disk (default: 5)
-C <num> Maximum number of non-leaf nodes in-memory cache (default: 1024)
-M <size> database memory cache size in MB (default: 100)
-I <file> save process PID to file (default value:/tmp/httpsqs.pid)
-A <auth> Access HTTPSQS authentication password (ex: mypass123)
-D Run as Daemon
-H Displays this help

Example:

Ulimit-shn 65535 httpsqs-d-P 1218-x/data0/queue

　　

Use the command "Killall Httpsqs", "Pkill Httpsqs", and "Kill ' Cat/tmp/httpsqs.pid '" to stop the HTTPSQS.

Note: Do not use the command "Pkill-9 Httpsqs" and "kill-9 Httpsqs process id" to end HTTPSQS, otherwise data that has not been saved to disk in memory will be lost.

　　5. HTTPSQS client uses the document:

　　(1), into the queue (put text messages in the queue):

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=put&data= URL-encoded text message &auth=mypass123 "

HTTP POST Protocol (for example, Curl command):

" URL-encoded text messages " " http://host:port/?name=your_queue_name&opt=put&auth=mypass123 "

Take the browser as an example:
　　

If the into queue succeeds, return:

Httpsqs_put_ok

If the in queue fails, return:

Httpsqs_put_error

If the queue is full, return:

Httpsqs_put_end

Starting with Httpsqs version 1.2, a line of "pos:xxx" is added to the HTTP header header returned to the client, outputting the read location point of the current queue, for example:

http/1.1 OK
Content-type:text/plain
keep-alive:120
Pos:19
Date:thu, 04:57:08 GMT
Content-length:14

Httpsqs_put_ok

　　(2), Out queue (remove text message from queue):

HTTP GET Protocol (for example, Curl command):

" http://host:port/?charset=utf-8&name=your_queue_name&opt=get&auth=mypass123 "  "http://host:port/?charset=gb2312&name=your_queue_name&opt=get&auth= mypass123"

Take the browser as an example:
　　

Returns the contents of the message queue to the client.

If there is no message queue that has not been fetched, it returns:

Httpsqs_get_end

Starting with Httpsqs version 1.2, a line of "pos:xxx" is added to the HTTP header header returned to the client, outputting the read location point of the current queue, for example:

http/1.1 OK
Content-type:text/plain; Charset=utf-8
keep-alive:120
Pos:7
Date:thu, 04:56:01 GMT
Content-length:18

Message Queuing content

　　Parameter charset description (for example:/?charset=utf-8):
Specifies the character encoding of the header of the HTTP output header, namely:
Content-type:text/plain; Charset=utf-8

Any character encoding registered in the IANA can be used, but not all browsers can parse all character encodings. For Chinese, the commonly used character encoding is: Utf-8, gb2312, GBK, GB18030, Big5 and so on.

　　(3), check the status of the queue (normal way, easy to view the browser):

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=status&auth=mypass123 "

Return (example):

HTTP Simple Queue Service v1.7
------------------------------
Queue Name:xoyo
Maximum Number of queues:1000000
Put Position of queue (1st lap): 45
Get Position of queue (1st lap): 6
Number of unread queue:39

If the queue write point value is greater than the maximum queue quantity value, the queue write point is reset to 1, and the new queue content is stored starting from 1, overwriting the contents of the original queue location point:

HTTP Simple Queue Service v1.7
------------------------------
Queue Name:xoyo
Maximum Number of queues:1000000
Put position of queue (2st lap): 4562
Get Position of queue (1st lap): 900045
Number of unread queue:104517

Take the browser as an example:
　　

　　(4), check the status of the queue (JSON way, easy for the program to process the return content):

This feature is supported starting with the HTTPSQS 1.3 release.

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=status_json&auth=mypass123 "

Return (example):

{"Name": "Xoyo", "Maxqueue": 1000000, "Putpos": 1, "Putlap": 6, "GetPos": 1, "Getlap": 39}

If the queue write point value is greater than the maximum queue quantity value, the queue write point is reset to 1, and the new queue content is stored starting from 1, overwriting the contents of the original queue location point:

{"Name": "Xoyo", "Maxqueue": 1000000, "Putpos": 4562, "Putlap": 2, "GetPos": 900045, "Getlap": 1, "Unread": 104517}

　　(5), view the contents of the specified queue location point:

Unlike a typical queuing system, HTTPSQS can view the contents of a specified queue ID (queue point), including the contents of a queue that is not out and out. It is easy to see if the content entered into the queue is correct.

Also, suppose you have a queue that sends SMS messages, the client daemon takes the information out of the queue, and calls the SMS Gateway interface to send SMS messages. However, if a period of time, "SMS Gateway Interface" has a fault, and this time queue location Point 300~900 information has been out of the queue, but send SMS failed, we can also be covered in the location point 300~900 before, see the contents of these location points, for the corresponding processing.

HTTP GET Protocol (for example, Curl command):

" http://host:port/?charset=utf-8&name=your_queue_name&opt=view&pos=5&auth=mypass123 "  "http://host:port/?charset=gb2312&name=your_queue_name&opt=view&pos=19& auth=mypass123"

POS >=1 and <= 1000000000

Returns the contents of the specified queue location point.

　　(6), reset the specified queue:

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=reset&auth=mypass123 "

If the reset succeeds, return:

Httpsqs_reset_ok

If the reset fails, return:

Httpsqs_reset_error

　　(7), change the maximum number of queues for the specified queue:

Default maximum queue Length (1 million): 1000000

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=maxqueue&num=1000000000&auth=mypass123 "

Num >=10 and <= 1000000000

If you change the maximum number of queues successfully, then return:

Httpsqs_maxqueue_ok

The maximum number of queues to change must be greater than the current queue write point. In addition, when the queue write point is less than the queue read point (that is, the put is in the second circle of the ring and get is in the first lap of the ring), the operation is canceled and then returned to the client with the following information:

Httpsqs_maxqueue_cancel

　　(8), without stopping the service, modify the time interval when the memory buffer contents are flushed to disk periodically:

This feature is supported starting with the HTTPSQS 1.3 release.

Default time interval: 5 seconds or httpsqs-s <second> parameter set value.

HTTP GET Protocol (for example, Curl command):

" http://host:port/?name=your_queue_name&opt=synctime&num=10&auth=mypass123 "

Num >=1 and <= 1000000000

If the modification interval succeeds, it returns:

Httpsqs_synctime_ok

If num is not between 1 and 1000000000, this operation will be canceled and then returned to the client with the following information:

Httpsqs_synctime_cancel

　　(9), password check failure:

This feature is supported starting with the HTTPSQS 1.5 release.

If the password check fails (/?auth=xxx), the following information is returned:

Httpsqs_auth_failed

　　(10), global error:

If a global error occurs (that is, an instruction, a parameter error, and so on), the following information is returned:

Httpsqs_error

　　6. HTTPSQS Client

(1), PHP client Documentation:

A, PHP Client Extension (third-party provided, for details, please visit: http://code.google.com/p/php-httpsqs-client/)

B, PHP client Class file (officially available: applicable to Httpsqs 1.7 or above, recommended. ）

View PHP Class source code: httpsqs_client.php

Example of using all functions of PHP Client: test_example.php

PHP Client command Line run Example: test_commandline.php

Usage:

<?php include_once ("httpsqs_client.php");      $httpsqs = new Httpsqs ($httpsqs _host, $httpsqs _port, $httpsqs _auth, $httpsqs _charset); /* 1. Put text information in a queue (note: If the PHP variable you want to put in the queue is an array, you need to use serialization, Json_encode, and other functions to convert to text beforehand) if the queue succeeds, returns a Boolean value: True if the in queue fails, returns a Boolean value: false */$re      Sult = $httpsqs->put ($queue _name, $queue _data); /* 2. Remove text information from a queue returns the contents of the queue if there are no queues that have not been fetched, the text message is returned: Httpsqs_get_end If an error occurs, returns a Boolean value: false */$result = $httpsqs->get (     $queue _name); /* 3. Remove text information from a queue and current queue read point POS Returns an array example: Array ("pos" = + 7, "data" = "text message") If there are no queues that have not been fetched, the array is returned: Array ("POS" =&G T    0, "Data" = "httpsqs_get_end") if an error occurs, returns a Boolean value: false */$result = $httpsqs->gets ($queue _name); /* 4.    View queue status (normal) */$result = $httpsqs->status ($queue _name); /* 5. View queue status (JSON) returns an example: {"name": "Queue_name", "Maxqueue": 5000000, "Putpos": "Putlap": 1, "GetPos":, "Getlap": 1, "     Unread ": Ten} */$result = $httpsqs->status_json ($queue _name); /* 6. View the contents of the specified queue location point backSpecifies the contents of the queue location point.     */$result = $httpsqs->view ($queue _name, $queue _pos); /* 7.     Resets the specified queue if the reset queue succeeds, returns a Boolean value: True if the reset queue fails, returns a Boolean value: false */$result = $httpsqs->reset ($queue _name); /* 8.    Change the maximum number of queues for a specified queue if the change succeeds, returns a Boolean value: True if the change operation is canceled, returns a Boolean value: false */$result = $httpsqs->maxqueue ($queue _name, $num); /* 9.  Modify the interval of time when the memory buffer contents are flushed to disk if the change succeeds, returns a Boolean value: True if the change operation is canceled, returns a Boolean value: false */$result = $httpsqs->synctime ($num);   ?>

　　

　　7, HTTPSQS production Environment typical application Case framework:

　　

A simple HTTPSQS client daemon framework, written in PHP, is as follows:

Environment, assuming PHP installation path is/usr/local/webserver/php, use PHP to write a file/opt/httpsqs_client_daemon.php:

<?php  include_once dirname (__file__). " /httpsqs_client.php ";     $httpsqs = new Httpsqs ($host, $port, $auth, $charset);  while (true) {    $result = $httpsqs->gets ($name);    $pos = $result ["POS"]; The read location point of the current queue message    $data = $result ["Data"];//The contents of the current queue message    if ($data! = "Httpsqs_get_end" && $data! = "Httpsqs_ ERROR ") {      ... To do the application operation ...    } else {      sleep (1);//Pause after 1 seconds, loop again    }  }  ?>

Under Linux, push to background execution:

nohup/usr/local/webserver/php/bin/php/opt/httpsqs_client_daemon.php 2>&1 >/dev/null &

Original: http://zyan.cc/httpsqs/

Lightweight open source simple Queue service based on HTTP protocol: httpsqs[to]