Summary of completing Port Programming in Windows

Completed the problem of connectex on the Port:

When the port is used on the client, you must note that you must first bind a port to the created socket (which is automatically allocated by default 0) and then throw it to the completed port before you can connect to ctex, otherwise (not bound), connectex will receive the 10022 error. At the same time, getpeername does not seem to support connectex, or other details may be set. I do not know for the time being, and the returned remote address is invalid (only valid for socket after Port completion is used for Server Accept return, it is also valid for the connet used for the customer ). However, it does not have much impact on the program. Since you have to actively connect to the remote address, you have long understood that you can save a copy of the remote address without getpeername. Note: The local address returned by getsockname is always valid.

The reason why getpeername cannot get the correct content is that after connectex returns, the socket-related attributes have not been updated (due to the characteristics of connectex ), you should call setsockopt to update the socket attributes and then call getpeername. The parameters are sol_socket and so_update_connect_context.

When the connectex function returns, the socket S is in the default state for a connected socket. the socket s does not enable previusly set properties or options until so_update_connect_context is set on the socket. use the setsockopt function to set the so_update_connect_context option.

For example:

Err = setsockopt (S, sol_socket, so_update_connect_context, null, 0 );

 

Complete port and ACE framework:

The ACE framework used varies depending on different platforms. For example, on a server in windows, the proactor framework is generally used in combination with various asynchronous operations, such as ace_asynch_acceptor/ace_asynch_connector. Because the internal implementation of proactor is the completion port, on the Windows platform, it is recognized that the best performance can be achieved.

If you use Linux, we recommend that you use the reactor framework + dev_poll_reactor to implement the reactor. This implementation uses the epoll mechanism, and the performance is superb.

The client generally uses reactor for compatibility consideration. Of course, if it is running on Windows, the default implementation is wfmo_reactor.

 

Iocp (I/O completion port, I/O completion port) is the best I/O model for scalability. This chapter will discuss the concept of port completion and its usage, introduce the architecture of a Scalable Server, and introduce the process of using iocp to design a Scalable Server program based on instances.

4.1 complete the port I/O model

When an application must manage multiple sockets at a time, the completion of the port model provides the best system performance. This model also provides the best scalability and is suitable for processing hundreds or thousands of sockets. Iocp technology is widely used in various types of high-performance servers, such as Apache. This section describes how to use it with a simple example.

4.1.1 what is a completion port object

The I/O completion port is a mechanism by which the application uses the thread pool to process asynchronous I/O requests. When processing multiple concurrent asynchronous I/O requests, using I/O to complete the port is faster and more effective than creating a thread in an I/O Request.

The initial design of the I/O completion port is that the application sends some asynchronous I/O requests. When these requests are completed, the device driver sorts these work items to the completion port, the thread pool waiting on the completion port can process the completion I/O. The completion port is actually a windows I/O structure. It can receive various object handles, such as file objects and socket objects. This section describes how to use the port model to manage sockets.

4.1.2 Method of Using iocp 1. Create a port object

To use the complete port model, call the createiocompletionport function to create a complete port object. Winsock uses this object to manage I/O requests for any number of socket handles. The function is defined as follows:

Handle createiocompletionport (handle filehandle,

Handle existingcompletionport, ulong_ptr completionkey, DWORD numberofconcurrentthreads );

Before explaining function parameters in detail, I will first introduce two different functions of this function.

(1) create a complete port object.

(2) associate one or more file handles (socket handles here) with an I/O completion port object.

When creating a port object, the unique parameter that needs to be set is numberofconcurrentthreads, which defines the number of threads that can be executed simultaneously on the port. Ideally, we want each processor to run only one thread to provide services for port completion to avoid thread context switching. The value of numberofconcurrentthreads is 0, indicating that the number of threads allowed by the system is as large as the number of processors. Therefore, you can simply use the following code to create a port object and obtain the handle of the port identified.

Handle hcompletion =: createiocompletionport (invalid_handle_value, 0, 0 );

2. I/O service thread and completion port

After the port object is successfully created, You can associate the socket handle with the object. Before associating a socket, you must first create one or more worker threads (I/O service threads ), execute and process the I/O requests sent to the completed port on the completed port. The key issue here is how many worker threads should be created. Note that the number of threads specified during port creation is not the same as the number of threads to be created. We recommend that you set the number of threads to the number of processors to avoid context switching. The numberofconcurrentthreads parameter of the createiocompletionport function explicitly tells the system the number of threads allowed to run simultaneously on the completed port. If more threads are created than numberofconcurrent threads, only numberofconcurrentthreads threads are allowed to run. But sometimes more threads need to be created, depending on the overall design of the program. If a thread calls a function, such as sleep or waitforsingleobject, and the function is suspended, one of the multiple threads starts to run, occupying the position of the sleeping thread. All in all, we always hope that there will be as many threads as specified by the createiocompletionport function in completing the I/O processing on the port. The final conclusion is that if you think the working thread will be congested (in the paused state), you should create more threads than the number specified by createiocompletionport.

With enough working threads to process the I/O requests on the port, the first three parameters of the createiocompletionport function are used.

L filehandle the socket handle to be associated

L existingcompletionport the port object handle created above

L completionkey specifies a unique (Per-handle) handle, which is associated with the filehandle socket handle. The application can store any type of information here, usually a pointer

The completionkey parameter is usually used to describe socket-related information. Therefore, it is called the unique (Per-handle) data of the handle. In the following example code, you can see its role.

3. Complete port and overlapping I/O

After associating a socket handle to a port, you can send overlapping messages and receive requests to process I/O. When these I/O operations are completed, the I/O system will send a complete notification packet to the completed port object. The I/O completion port queues these packets in FIFO mode. The application can use the getqueuedcompletionstatus function to obtain packets in these queues. This function should be called in the service thread that processes the completed object I/O.

Bool getqueuedcompletionstatus (

Handle completionport, // complete the port object handle

Lpdword lpnumberofbytes, // get the number of bytes transmitted during the I/O operation

Pulong_ptr lpcompletionkey, // obtain the unique data of the specified handle when associating the socket

Lpoverlapped * lpoverlapped, // gets the overlapped structure specified during the shipping I/O operation

DWORD dwmilliseconds // If the completion port does not complete the packet, this parameter specifies the waiting event, infinite is infinite

);

The I/O service thread calls the getqueuedcompletionstatus function to obtain information about the socket where an event occurs. The number of bytes transmitted is obtained through the lpnumberofbytes parameter, the lpcompletionkey parameter is used to obtain the unique (Per-handle) data associated with the socket. The lpoverlapped parameter is used to obtain the overlapping object address used for shipping I/O requests, obtain the unique I/O (Per-I/O) data.

Among these parameters, the most important is the per-handle data and per-I/O data.

The lpcompletionkey parameter contains the data called Per-handle, because when the socket is first associated with the completion port, the data is associated with a socket handle. This is the completionkey parameter passed to the createiocompletionport function. As mentioned above, you can pass data of any type to this parameter.

The lpoverlapped parameter points to an overlapped structure. The structure is followed by the data we call per-I/O. This can be any information the worker thread wants to know when processing packets.

4.1.3 sample program

The following is a simple example of a TCP server that uses the iocp model. It only prints the data received from the client. The high-performance and Scalable Server class ciocpserver will be designed based on this example later.

There are two types of threads in this example: the main thread and the thread it creates. The main thread creates a listening socket, creates additional working threads, associates with iocp, and waits for and accepts incoming connections. The thread created by the main thread is responsible for processing I/O events. These threads call the getqueuedcompletionstatus function to complete the I/O operations on the completed port object.

After the getqueuedcompletionstatus function is returned, it indicates that one of the following events has occurred.

(1) If the call to getqueuedcompletionstatus fails, an error occurs on this socket.

(2) If bytestransferred is 0, the socket is disabled by the other party. Note that the per-handle data is used to reference sockets related to I/O operations.

(3) the I/O request is successfully completed. You can use the operationtype field in the per-I/O data (this is the custom structure of the program) to check which I/O request has been completed.

The program first defines the structure types of the per-handle data and per-I/O operation data.

// Initialize the Winsock Library

Cinitsock thesock;

# Define buffer_size 1024

Typedef struct _ per_handle_data // per-handle data

{

Socket s; // corresponding socket handle

Sockaddr_in ADDR; // customer address

} Per_handle_data, * pper_handle_data;

Typedef struct _ per_io_data // per-I/O data

{

Overlapped ol; // overlapping structure

Char Buf [buffer_size]; // data buffer

Int noperationtype; // Operation Type

# Define op_read 1

# Define op_write 2

# Define op_accept 3

} Per_io_data, * pper_io_data;

The main thread first creates the completed port object, creates a working thread to process the events in the completed port object, and then creates a listening socket to start listening to the Service port. Then it enters an infinite loop, process the incoming connection request as follows:

(1) Call the accept function to accept pending connection requests.

(2) After receiving the new connection, create a per-handle data for it and associate them with the completion port object.

(3) deliver a receiving request on the new socket. The worker thread is responsible for processing the I/O.

The following is the specific implementation code.

Void main ()

{

Int nport = 4567;

// Create a completed port object and create a working thread to process events in the completed port object

Handle hcompletion =: createiocompletionport (invalid_handle_value, 0, 0 );

: Createthread (null, 0, serverthread, (lpvoid) hcompletion, 0, 0 );

// Create a listening socket, bind it to a local address, and start listening

Socket slisten =: socket (af_inet, sock_stream, 0 );

Sockaddr_in Si;

Si. sin_family = af_inet;

Si. sin_port =: ntohs (nport );

Si. sin_addr.s_un.s_addr = inaddr_any;

: BIND (slisten, (sockaddr *) & Si, sizeof (SI ));

: Listen (slisten, 5 );

// Cyclically process incoming connections

While (true)

{// Wait to accept pending connection requests

Sockaddr_in saremote;

Int nremotelen = sizeof (saremote );

Socket snew =: accept (slisten, (sockaddr *) & saremote, & nremotelen );

// After receiving the new connection, create a per-handle data for it and associate them with the completion port object.

Pper_handle_data pperhandle =

(Pper_handle_data): globalalloc (gptr, sizeof (per_handle_data ));

Pperhandle-> S = snew;

Memcpy (& pperhandle-> ADDR, & saremote, nremotelen );

: Createiocompletionport (handle) pperhandle-> S, hcompletion, (DWORD) pperhandle, 0 );

// Deliver a receiving request

Pper_io_data pperio = (pper_io_data): globalalloc (gptr, sizeof (per_io_data ));

Pperio-> noperationtype = op_read;

Wsabuf Buf;

Buf. Buf = pperio-> Buf;

Buf. Len = buffer_size;

DWORD dwrecv;

DWORD dwflags = 0;

: Wsarecv (pperhandle-> S, & Buf, 1, & dwrecv, & dwflags, & pperio-> ol, null );

}

}

The I/O service thread cyclically calls the getqueuedcompletionstatus function to remove the completed I/O packets from the I/O completed port, and then processes them according to the packet type. The specific program code is as follows:

DWORD winapi serverthread (lpvoid lpparam)

{// Get the finished port object handle

Handle hcompletion = (handle) lpparam;

DWORD dwtrans;

Pper_handle_data pperhandle;

Pper_io_data pperio;

While (true)

{// Wait for I/O to be completed on all sockets associated with the completion port

Bool Bok =: getqueuedcompletionstatus (hcompletion,

& Dwtrans, (lpdword) & pperhandle, (lpoverlapped *) & pperio, wsa_infinite );

If (! Bok) // an error occurs on this socket

{: Closesocket (pperhandle-> S );

: Globalfree (pperhandle );

: Globalfree (pperio );

Continue;

}

If (dwtrans = 0 & // The socket is disabled by the other party

(Pperio-> noperationtype = op_read | pperio-> noperationtype = op_write ))

{: Closesocket (pperhandle-> S );

: Globalfree (pperhandle );

: Globalfree (pperio );

Continue;

}

Switch (pperio-> noperationtype) // you can use the noperationtype field in the per-I/O data to check what I/O requests have been completed.

{

Case op_read: // complete a request

{Pperio-> Buf [dwtrans] = '/0 ';

Printf (pperio-> BUF );

// Continue delivery to receive I/O requests

Wsabuf Buf;

Buf. Buf = pperio-> Buf;

Buf. Len = buffer_size;

Pperio-> noperationtype = op_read;

DWORD nflags = 0;

: Wsarecv (pperhandle-> S, & Buf, 1, & dwtrans, & nflags, & pperio-> ol, null );

}

Break;

Case op_write: // in this example, these types of I/O requests are not shipped.

Case op_accept:

Break;

}

}

Return 0;

}

4.1.4 properly disable iocp

The example in section 4.1.3 does not cover how to properly close the I/O completion port, especially when multiple threads execute I/O on the socket. The main thing to avoid is to release its overlapped structure when overlapping operations are in progress, the best way to prevent this from happening is to call the closesocket function on each socket handle-all pending overlapping I/O operations will be completed. Once all socket handles are closed, it is time to terminate the worker threads that process I/O on the port. This can be achieved by using the postqueuedcompletionstatus function to send a specific completion packet to the working thread. This completion packet notifies the working thread to exit immediately. The postqueuedcompletionstatus function is defined as follows:

Bool postqueuedcompletionstatus (

Handle completionport, // complete the port object handle

DWORD dwnumberofbytestransferred, // specify

// Return value of the lpnumberofbytestransferred Parameter

Ulong_ptr dwcompletionkey, // specify the return value of the lpcompletionkey parameter of the getqueuedcompletionstatus Function

Lpoverlapped // specify the return value of the lpoverlapped parameter of the getqueuedcompletionstatus Function

);

When the worker thread receives three parameters of getqueuedcompletionstatus, it can decide whether to exit. For example, you can use closehandle to close the completion port after passing 0 to the dwcompletionkey parameter and all worker threads exit.

The finished port is the best I/O model in terms of performance and scalability so far. There is no limit on the number of sockets associated with the completed port object. Only a small number of threads are required to process the completed I/O. This chapter will discuss how to use the port to develop Scalable high-performance server programs.