Webtail file reading, file monitoring, WebSocket

file Read

To introduce the background, before the server on the need for a long time tail a log, before often through a terminal connected to the server, but for a long time to observe, it is not convenient: always have to open the terminal, can not use mobile phones and other mobile devices to view; many people share to see more trouble, all need to log on to the server.

Webtail implementation is similar to tail, can continuously read a file, and the contents of the file through WebSocket, real-time push to the web side. Webtail files are read based on Linux inotify, so there is no portability, WebSocket uses ASIO based WEBSOCKETPP, code maintenance here.

Webtail code maintenance in Https://gitorious.org/webtail/webtail, is currently a simple and usable small application, there are many places to upgrade.

This article describes the section under which the file is read.

File reads are divided into two parts, file reading and file monitoring.

File reads:
Because UNIX files open standalone Maintenance file table multiple times, shared V-node table (reference apue 3.10 file sharing). As a result, only one file descriptor and current file offset are maintained, and each read gets the current size of the document by Fstat, compared to the maintained file offset and, if less than the file size, reads the file until the end and outputs it.
Here are a few things to imitate TAILF:

The first is to not modify the file access time. This is done by adding o_noatime in the open system call, as explained by man Open: does not update the file last access time (St_atime in the Inode) when the "file is" Read (2). This flag is intended to use by indexing or backup programs, where it use can significantly reduce the amount of disk AC Tivity. This flag may is effective on all file systems. One example is NFS, where the server main?tains the access time. This parameter is started, and the atime is not modified when the file is read by the read call.

Second is the first read, read the last 10 lines of the file (because WebSocket sent without caching, so from the web side can not see). The code here also refers to the TAILF:

The code is as follows

Copy Code

Char *buffer = new Char[initlen * Bufsiz]; char *p = buffer; Char Linebuffer[bufsiz]; int readsize; int linecount = 0; BOOL head = TRUE; int i = 0, j = 0; while ((ReadSize =:: Read (FD, Linebuffer, BUFSIZ-1)) > 0) { for (i = 0, j = 0; i < readsize; ++i) { Read line, save to buffer if (linebuffer[i] = = ' n ') { std::memcpy (P, Linebuffer + J, i-j + 1); Std::memset (p + i-j + 1, ', 1); j = i + 1; if (++linecount >= initlen) { LineCount = 0; Head = FALSE; } p = buffer + (LineCount * bufsiz); } } Read break in the middle of line if (J < i) { Finished read all files if (ReadSize < Bufsiz) { std::memcpy (P, Linebuffer + J, i-j + 1); Std::memset (p + i-j + 1, ', 1); ++linecount; else if (j = = 0) {//Long line drop? Continue } else { Not finished, seek to line begin CurPos = Lseek (FD, J-i-1, seek_cur); } } } Std::string Initreadresult; if (head) { for (i = 0; i < LineCount; ++i) { Initreadresult + = (buffer + i * bufsiz); } } else { for (i = LineCount i < Initlen; ++i) { Initreadresult + = (buffer + i * bufsiz); } for (i = 0; i < LineCount; ++i) { Initreadresult + = (buffer + i * bufsiz); } } CurPos = Lseek (FD, 0, seek_cur); Delete buffer;

First declare a cache to hold the final n rows, the longest line length is bufsiz (this is defined in Linux as 8192 bytes), and then read BUFSIZ-1 bytes at a time (the last one to put, like Fgets implementations), Resolves the newline character (only n) because it is intended to be used only in Linux. Finally, read the last n rows of data from the cache to string based on the state.

If it is not an initial read, the previous logic has been made, comparing the file size obtained by Fstat to the current offset of the saved, and if there is a new content, read it and return it directly.


File Monitoring


File monitoring directly through the Linux INotify interface implementation. No portability is considered here, and there is nothing like TAILF to determine whether or not to support inotify, or, if not, to read it using a circular rotation.

The use of inotify or more convenient is basically: Inotify_init,inotify_add_watch, and then with the read system call, get file modification information. So the implementation is also very convenient.

The first is to initialize the INotify within the constructor:

INOTIFYFD = Inotify_init ();

It then provides a watch interface to add monitoring and callbacks to the corresponding file by passing in the callback function read in the Tfile object and content described previously.

The code is as follows

Copy Code

void Filewatcher::watch (boost::shared_ptr< tfile > Tfile, std::list< filewatcher::readcallback > Callbacklist) { if (!tfile->haserror () &&!callbacklist.empty ()) { int WD = Inotify_add_watch (INOTIFYFD, Tfile->name (). C_STR (), in_modify); if (wd > 0) { Tfilemap.insert (Std::make_pair<int, boost::shared_ptr<tfile> > (wd, Tfile)); Callbackmap.insert (Std::make_pair<int, std::list<readcallback> > (wd, Callbacklist)); Init read std::string initcontent = Tfile->read (); Boost_foreach (Readcallback &callback, callbacklist) { Callback (Initcontent); } } } }

This is where the Modify event for adding the file is registered to the kernel through the tfile filename, and after the registration is successful, the initial read (there is a small problem, because there is no cache on the back of the websocket end, so there is no WebSocket client connection due to the initial read, So the original content cannot be read through the Web, which is the last 10 lines of the file. At the same time, this class maintains two hashmap, the Listener descriptor Wd->tfile and wd->callbacklist respectively.

After listening, is to start listening, that is, through the reading of FD, aware of the change of the listening file, because this only listens to file modifications, then read this event, you can read the file for incremental reading (previously described the Reading method).

The code is as follows

Copy Code

char * buffer = new Char[inotifybuffersize]; while (!_interrupted) { if (read (INOTIFYFD, buffer, inotifybuffersize) < 0) { if (errno = = eintr) { Interrupt Delete buffer; Return } } struct Inotify_event *event = (struct inotify_event *) buffer; int WD = event->wd; Boost_auto (Tfileiter, Tfilemap.find (wd)); if (Tfileiter!= tfilemap.end ()) { boost::shared_ptr<tfile> Tfile = tfileiter->second; std::string content = Tfile->read (); Boost_auto (ITER, Callbackmap.find (wd)); if (ITER!= callbackmap.end ()) { std::list<readcallback> callbacks = iter->second; Boost_foreach (Readcallback &callback, callbacks) { Callback (content); } } } } Delete buffer;

This refers to the INotify document, which first reads the buffer size set to

static const int inotifybuffersize = sizeof (struct inotify_event) + Name_max + 1;

That is, the length of the inotify_event structure, and the maximum name. Since inotify_event is a variable-length field (containing an optional file name segment), the system-restricted file name maximum is used here Name_max, which is defined in climits and is 255 bytes in Linux.
Read is then invoked through the system, reading the file descriptor INOTIFYFD, where if no new events are generated, read will enter a blocking state, saving system resources. If there is a return, the returned Inotify_event object is processed (note that there is no filename when listening for the Modify event). By the WD in the structure, the corresponding Tfile object is fetched from the previously saved HashMap, then the corresponding callback function is read, and the read content is returned.
There is a small problem to be dealt with, that is, how to interrupt the read. In order to be able to test through unit tests in Gtest, you can tell by looking at the manual that if the read call is interrupted by a system signal, the error code will be labeled Eintr. Therefore, when the reading fails, you can check the errno to determine if the signal is interrupted.

Because the program will run all the time and know that the signal is terminated, the destructor is not very important. Here the destructor is removed by calling Inotify_rm_watch all the previously saved WD, and then closed by calling the INotify file descriptor via close call:

code is as follows	copy code
filewatcher::~filewatcher () {     if (inotifyfd > 0) {         Boost::unordered::unordered_map<int, boost::shared_ptr< Tfile> >::iterator iter;         for (iter = Tfilemap.begin (); Iter!= tfilemap.end (); ++iter) {             Inotify_rm_watch (INOTIFYFD, Iter->first);        }                Close (INOTIFYFD);    } }

WebSocket

Describes how to listen and incrementally read files on the server side, where a simple websocket service is implemented through a WEBSOCKETPP based on boost ASIO , and can communicate with the browser, will read the file through the WebSocket protocol for real-time transmission.

For a brief introduction to WebSocket, refer to Wikipedia. WebSocket protocol, defined in rfc6455.

WEBSOCKETPP The WebSocket and simple HTTP are better encapsulation, as long as the implementation of a few handler, you can complete the connection, message, and other operations and control. The main need to deal with, may have the following handler:

code is as follows	copy code
typedef Lib:: Function<void (CONNECTION_HDL) > Open_handler; typedef lib::function<void (CONNECTION_HDL) > Close_handler; typedef lib::function<void (CONNECTION_HDL) > Http_handler; typedef lib::function<void (CONNECTION_HDL,MESSAGE_PTR) > Message_handler

Handle connection creation, connection shutdown, HTTP requests, and message requests separately. Where CONNECTION_HDL is the weak_ptr of the connection.

This is a simple use of websocket, the only requirement is to maintain the established connection (when creating the connection record, when the connection is removed), and then by registering its callback into the file monitoring class, push the message to the WebSocket client in real time.

First, maintain a set to hold all the connections that are currently established:

typedef std::set<websocketpp::connection_hdl,boost::owner_less<websocketpp::connection_hdl> > ConnectionSet;

Then insert into this set when the connection is established:

The code is as follows	Copy Code
void Websocketserver::onopen (WEBSOCKETPP::CONNECTION_HDL HDL) { boost::lock_guard<boost::mutex> Lock (_mutex); _conns.insert (HDL); }

When the connection is closed, remove the connection:

The code is as follows	Copy Code
void Websocketserver::onclose (WEBSOCKETPP::CONNECTION_HDL HDL) { boost::lock_guard<boost::mutex> Lock (_mutex); _conns.erase (HDL); }

Also, define a callback that lets the Filewatcher call:

code is as follows	copy code
void Websocketserver::write (const std::string& content) {     boost::lock_guard<boost:: mutex> Lock (_mutex);     Boost_auto (it, _conns.begin ());     for (; it!= _conns.end (); ++it) {         _s.send (*it, Content, Websocketpp::frame::opcode::text);    } }

This callback is very simple, that is, when there is read to the content, traversing the connection collection, to each connection to send specific content.

Finally, to facilitate user access, when discovery is a standard HTTP request, we return a simple HTML for displaying and establishing websocket connections. If a specific HTTP_HANDLER,WEBSOCKETPP is not specified, an HTTP 426 error (Upgrade Required) is returned when the discovery request is HTTP.

The code is as follows	Copy Code
void Websocketserver::httphandler (WEBSOCKETPP::CONNECTION_HDL HDL) { Server::connection_ptr connptr = _S.GET_CON_FROM_HDL (HDL); Connptr->set_status (Websocketpp::http::status_code::ok); Connptr->set_body (htmlcontent); }

In addition, there is a small pit. According to WEBSOCKETPP's example, the Listen function that invokes the server directly at startup, and uses the implementation of only the port number. The actual use of the process, found that only the implementation of the port number, directly using the IPv6 protocol. Although if this machine supports both IPv6 and IPv4, the ports of the two protocols will listen, but when the server shuts down the IPv6, it will cause boost ASIO to throw the address_family_not_supported exception, resulting in the application being forced out. To be compatible in this way, the exception is crawled, and the attempt to IPv4 the protocol is degraded, which is good for use on only IPv4 servers.

code is as follows	copy code
try{     _s.listen (port); } catch (Boost::system::system_error const& e) {     if (e.code () = = Boost::asio::error:: address_family_not_supported) {         _s.listen (boost::asio::ip::tcp::v4 (), port);    } } catch (...) {     throw; }