Cross-platform network communication and Server programming framework Library (ACL Library)

I. Description

The ACL Project is a cross-platform network communication library and Server programming framework that supports Linux, Win32, Solaris, MacOS, And FreeBSD. It also provides more practical function libraries. With this library, you can easily write server programs, web applications, and database applications that support multiple modes (multithreading, multi-process, non-blocking, trigger, and UDP. In addition, this library provides client communication libraries for common applications (such as HTTP, SMTP, ICMP, memcache, and beanstalk) and common stream codec libraries: XML/JSON/MIME/base64/uucode/qpcode/rfc2047 etc.

1.1 library Composition

This project mainly contains five databases and a large number of examples. The five databases are described as follows:

• 1) lib_acl: This library is the most basic library, and the other four libraries depend on this library. This library is implemented in C language.
• 2) lib_protocol: This library mainly implements the HTTP protocol and ICMP/ping protocol. This library is implemented in C language.
• 3) lib_acl_cpp: This library encapsulates lib_acl/lib_protocol libraries in the C ++ language, and adds some other valuable functional applications.
• 4) lib_dict: This library mainly implements the key-value dictionary repository, which also depends on the database bdb, CDB, and tokyocabinet.
• 5) lib_tls: This library encapsulates the OpenSSL library so that the communication mode of lib_acl can support SSL.

1.2 functional modules 1.2.1 network communication library

• Stream processing module (acl_vstream): This module is the most basic stream communication module for the entire ACL network communication. It not only supports Network streams, but also file streams. It mainly supports:
• 1. Read data by row. compatible with \ r \ n under Win32 and with \ n Terminator under UNIX
• 2. Read data by row but remove \ n or \ r \ n at the end automatically
• 3. Read data using a string Separator
• 4. Read data of the specified length
• 5. Retry to read a row of data or the specified length of the attempted read data
• 6. Network I/O status detection
• 7. Write a row of data
• 8. Write Data by format, similar to fprintf.
• 9 file stream location operations, similar to fseek
• 10 write a group of data at a time, similar to writev in UNIX
• 11. truncates the file, similar to ftrunk.
• 12. Obtain the file size.
• 13. Get the current file stream pointer location, similar to ftell.
• 14. Obtain the file size

• 15. Obtain the local address and remote address of the network stream.

• Network Operation Module: This module mainly supports network server listening (TCP/UDP/Unix domain Interface) and network client connection (TCP/Unix domain interface) DNS domain name query and result caching (support for calling the system's gethostbyname function and direct transmission of DNS protocol), set interface (socket) operations, and local Nic retrieval.

• Non-blocking network streams: Supports non-blocking connection, read (read by row, read by specified length), write (write row, write specified length, write a set of data) and other operations.

• Common network application protocol Library (lib_protocol/lib_acl_cpp): Supports common network application protocols, such as HTTP, SMTP, and ICMP. The HTTP and ICMP modules implement blocking and non-blocking communication methods. In addition, the HTTP protocol also supports two communication modes: server and client in lib_acl_cpp of C ++. When used as a server, it supports interfaces similar to Java httpservlet, when used as a client, the connection pool and cluster management modes are supported, this module also supports cookie, session, http mime file upload, multipart transmission, automatic character set conversion, automatic decompression, resumable upload, and many other functions.

• Common network communication Libraries: Supports the communication libraries of memcached, beanstalk, and handler socket clients. The communication libraries support connection pools.

1.2.2 network I/O event Engine

Supports select (Unix/Linux/Win32), Poll (Unix/Linux), epoll (Linux), kqueue (BSD), devpoll (Solaris), and iocp (win32) windows message (win32) and other system event engines. Both the reactor and proactor programming models are supported.

1.2.3 network server framework

This module is the most important module in the ACL and provides the infrastructure commonly used in Server programming. The server framework comes from the famous Postfix and has made many extensions based on it, currently, the following major service models are supported:

• 1. Multi-Process Model:A process is connected. The advantage of this model is that the programming is simple, secure, and stable. The disadvantage is that the concurrency is not high;
• 2 multi-process and multi-thread model:Each sub-process processes all client connections by a group of threads in the thread pool. The I/O event is triggered. The connection is bound to a thread only when the connection has data readable, the thread is immediately returned to the thread pool after processing. The biggest advantage of this model is that a small number of threads can be used to process a large number of client connections, and the programming is relatively simple (compared with the non-blocking model );
• 3 multi-process non-blocking model:Each sub-process is composed of a separate non-blocking thread that processes a large number of external client connections (similar to nginx/squid/IRCD) in fully non-blocking Io mode ), the advantage of this model is that the processing efficiency is high and the resource usage is small. It can process a large number of client connections, but the programming is complicated;
• 4. UDP Communication Model:This model is a service model added to support udp Network processes;
• 5. Producer model:The instance of this model is mainly used to process the background service process of some scheduled tasks (similar to the crontab of the system ).

The sub-processes in the server framework use the semi-resident service model, supports the sub-process pre-start mechanism, maximum and minimum number of processes, sub-process exception alerts, and listening to multiple addresses for a single process (TCP/UDP sockets and Unix domain sockets can be monitored at the same time) sub-process security control, log output to syslog-ng, multi-process TCP connection uniformity; adopt configuration file drive mode, each service has a configuration file, it facilitates process management and online upgrading of service processes.

1.2.4 Common Data Structure Module

This module provides an array structure such as a common hash table (and multiple hash algorithms), a dynamic array, a two-way linked list, a balanced binary tree, a queue, a binary search tree, And a 256 cross matching tree; provides a unified data structure Traversal method (using acl_foreach ).

1.2.5. Memory Operation Module

This module provides three memory pool models:

• 1. Basic memory mode:The system's malloc/free API is encapsulated internally to provide security measures such as memory verification. This method also provides an external registration interface, allowing users to register their own memory allocation models;
• 2-chip memory (slab) mode:Multiple fixed-length memory chains are allocated based on different sizes, which can effectively reduce memory fragments and greatly improve memory allocation efficiency;
• 3. Memory Chain Mode:Allocating small memory with different lengths on one memory page can effectively use the memory and reduce memory waste.

1.2.6. common string operation modules

Supports string matching, forward/backward comparison, string segmentation, string case-sensitive conversion, H2B/B2H conversion, URL encoding/decoding, and other functions.

1.2.7. File directory Module

Supports multi-level Directory Creation, multi-level directory scanning, file handle caching, and other functions. At the same time, the multi-level directory is processed in a circular manner, avoiding the potential risk of stack overflow in recursive mode.

1.2.8. Read the configuration file module

Supports reading configuration files in the form of name = value. If the value is long, you can use a backslash () to fold rows and use a configuration table to extract configuration items in the configuration file.

1.2.9. Thread and Thread Pool Module

Provides cross-platform support for POSIX standard thread interfaces (Win32 supported ); the thread pool module minimizes lock conflicts during thread task allocation through multiple measures (used in the multi-process multi-thread service model in the ACL Server framework ).

K-V disk storage module for numeric keys (zdb)

The key/Value block storage method is used. Because the key is limited to the number type, you only need to calculate the key location in the memory. The key stores the value location, therefore, you only need to locate the disk twice for any data query. (I am in Hexun traffic statistics storage, the use of this method to replace the BSD, TC, etc. using B tree K-V storage)

1.2.11. Stream parsing Database

This module includes codec libraries such as XML, JSON, rfc2047, mime, base64, uucode, qpcode, and charset, all of which adopt stream parsing, suitable for multiple network I/O models (blocking/non-blocking TCP and UDP ).

1.2.12. Database encapsulation Library

A unified database operation interface and connection pool processing method are designed. Currently, SQLite/MySQL is supported.

1.2.13. File queue processing database

Allows you to create, scan, and delete multiple directory queue files. It is often used in service programs for temporary file queue scheduling.

2. platform support and compilation

The entire project currently supports Linux (as4, 5, 6, cs4, 5, 6, UBUNTU), Windows, MacOS, FreeBSD, and Solaris.

• Linux/Unix: the GCC compiler directly enters the lib_acl/lib_protocol/lib_acl_cpp/lib_dict/lib_tls directory in the command line mode of the terminal, and runs the make command.
• Windows: You can use vc2003, vc2008, vc2010, and vc2012 for compiling. (If you need to compile with vc6/vc2005, you can refer to the compiling conditions of vc2003 ).

When using dynamic libraries in Win32 environments, pay attention to the following points:

• When using a dynamic library of lib_acl, you must predefine it in your project: acl_dll;
• When using the HTTP library or ICMP library in the lib_protocol dynamic library, you must pre-define http_dll or icmp_dll in the project;
• When using the dynamic library of lib_acl_cpp, You need to pre-define acl_cpp_dll in the project. If you use the vc2003 compiling environment, you also need to predefine vc2003;
• When using a dynamic library of lib_dict, you must pre-define dict_dll in the project;
• When using a dynamic library of lib_tls, you must pre-define tls_dll in the project.
  

Iii. Project materials

GitHub:Https://github.com/zhengshuxin/acl

SourceForge:Http://sourceforge.net/projects/acl/

Related Technical blog:Http://zsxxsz.iteye.com/

QQ group:242722074

The ACL project has a large number of test cases (nearly 200): https://github.com/zhengshuxin/acl/blob/master/SAMPLES.md

In addition, there are several practical tools:

1. server program generation Wizard: Ghost (or even HTTP web server program)

2. Globally Unique id generator: https://github.com/zhengshuxin/acl/tree/master/app/gid

3. Network status monitoring tool (win32): https://github.com/zhengshuxin/acl/tree/master/app/net_tools

4. TCP connection distributor: https://github.com/zhengshuxin/acl/tree/master/app/master_dispatch

Character Set converter: https://github.com/zhengshuxin/acl/tree/master/app/jencode

4. Example 1: category index chart:

2. asynchronous Io class inheritance Diagram

3. Io stream inheritance Diagram

4. cgi program for receiving uploaded files:

5. File Download client program:

6. webpage download program: