Multi-threaded RPC in Linux

When RPC is used, rpcgen-m in Linux cannot generate RPC proxy stubs with multiple threads, but the generated code has the thread security feature. Some netizens on the network have discussed an implementation method, that is, to create a thread to process client code when receiving client requests.

A foreign friend gives the implementation code (http://www.redhat.com/archives/open-source-now-list/2004-June/msg00000.html), but it doesn't run successfully when copied to my Linux (Linux rhl5. In China, an author also writes an example

UNIX network programming first volume to find the square root of the sample written URL is as follows: http://blog.chinaunix.net/u1/37472/showart_726114.html
I still failed to run (RPC: Procedure unavailable) using this example. by increasing the delay of the main thread, I got the correct operation.
However, by increasing the latency, the RPC server will only process data for a slow time. If the time is too long, RCP calls will time out. If the time is too short, RCP calls (new threads) the problem still persists. I use a local latency of one second and call 100 times at a certain time interval.

The current server has a segment error.
I changed the pointer to a variable and encountered an error: RPC: server can't decode arguments. Later, I found that the structure of the RPC call parameter is not a level-1 pointer, but a multi-level pointer. That is to say, the structure is still a pointer variable member.

After testing, I summarized the time sequence of RPC calls:
Svc_run ()
|-> Get customer request
| Create parameters for customer data
| Call server stubs (non-multithreading)
| Create a new thread to replace the server's Stub execution body | the new thread starts.
| Thread creation completed | access parameters
| <---- Clear parameters | access error (data has been cleared)
 
The time sequence after the insert time wait is:
Svc_run ()
|-> Get customer request
| Create parameters for customer data
| Call server stubs (non-multithreading)
| Create a new thread to replace the server's Stub execution body | the new thread starts.
| Thread creation completed | access parameters
| Waiting... | access the user server's execution body
| <----- Clear parameters | execute body Processing
| .........
| Return Client
Demonstrate the call sequence after obtaining the wait time, so that the call can be successful. If the client requests continuously and the client access parameters are not completed, the problem may occur. The time is related to the server's processing capability and Running Load.

I thought of using a thread lock to solve the problem of waiting. After 100 consecutive calls were generated, no problem was found, but it does not mean no problem, i'm just guessing about the RPC running process, and I don't know whether the original status information can be returned normally when I return it to the customer.

? What if the returned data is not a simple value but a struct data? So I hope to see my original friend. If something goes wrong during use, please let me know. I will also test it when I have time.

The following is the server code, which is generated by rpcgen and then modified.

/* <Br/> * Please do NOT edit this file. <br/> * it was generated using rpcgen. <br/> */<br/> # include <unistd. h> <br/> # include "Square. H "<br/> # include <stdio. h> <br/> # include <stdlib. h> <br/> # include <rpc/pmap_clnt.h> <br/> # include <string. h> <br/> # include <memory. h> <br/> # include <sys/socket. h> <br/> # include <netinet/in. h> </P> <p> # ifndef sig_pf <br/> # define sig_pf void (*) (INT) <br/> # endif </P> <p> P Thread_t p_thread; <br/> pthread_attr_t ATTR; </P> <p> # ifdef using_mt_rpc_with_lock <br/> pthread_mutex_t rpcmutex; <br/> # endif </P> <p> void * <br/> square_prog_thread_func (void * PARAM ); </P> <p> # ifdef complete <br/> void <br/> square_prog_2_thread_func (<br/> struct svc_req * rqstp, <br/> Register svcxprt * transp, <br/> pthread_mutex_t * pmutex); <br/> # else <br/> void <br/> square_prog_2_thread_func (< Br/> struct svc_req * rqstp, <br/> Register svcxprt * Transp ); <br/> # endif </P> <p> static void <br/> square_prog_2 (struct svc_req * rqstp, register svcxprt * Transp) <br/>{< br/>/* square_prog_2_thread_func (rqstp, Transp); <br/> return; */<br/> struct data_str <br/>{< br/> struct svc_req * rqstp; <br/> svcxprt * Transp; <br/> # ifdef using_mt_rpc_with_lock <br/> pthread_mutex_t * pmutex; <br/> # endif <br/>} * dat A_ptr = (struct data_str *) malloc (sizeof (struct data_str); <br/> If (data_ptr = NULL) {<br/> # ifdef success <br/> square_prog_2_thread_func (rqstp, transp, null); <br/># else <br/> square_prog_2_thread_func (rqstp, Transp ); <br/> # endif <br/> return; <br/>}< br/> data_ptr-> rqstp = rqstp; <br/> data_ptr-> Transp = Transp; <br/> # ifdef using_mt_rpc_with_lock <br/> data_ptr-> pmutex = & rpcmutex; <Br/> pthread_mutex_lock (& rpcmutex); <br/> # endif <br/>/* Create a New thread for client. */<br/> pthread_attr_setdetachstate (& ATTR, pthread_create_detached); <br/> pthread_create (& p_thread, & ATTR, square_prog_thread_func, (void *) data_ptr ); <br/> # ifdef using_mt_rpc_with_lock <br/> pthread_mutex_lock (& rpcmutex); <br/> pthread_mutex_unlock (& rpcmutex ); <br/> # else <br/> sleep (1); <br/> # endif <br/>}</P> <p> Void * <br/> square_prog_thread_func (void * PARAM) <br/> {<br/> struct svc_req * rqstp <br/> {<br/>; <br/> svcxprt * Transp; <br/> # ifdef using_mt_rpc_with_lock <br/> pthread_mutex_t * pmutex; <br/> # endif <br/>} * ptr_data; </P> <p>/* recover Param from thread Param */<br/> ptr_data = (struct thr_data *) Param; <br/> struct svc_req * rqstp = (ptr_data-> rqstp); <br/> Register svcxprt * Transp = (PTR _ Data-> Transp); <br/>/* continue server processing... */<br/> # ifdef feature <br/> square_prog_2_thread_func (rqstp, transp, ptr_data-> pmutex); <br/> # else <br/> reverse (rqstp, transp); <br/> # endif <br/> free (PARAM); <br/> return NULL; <br/>}</P> <p> # ifdef using_mt_rpc_with_lock <br/> void <br/> square_prog_2_thread_func (<br/> struct svc_req * rqstp, <br/> Register svcxprt * Transp, <br/> pthread_mutex_t * pmutex) <br/> # else <br/> void <br/> square_prog_2_thread_func (<br/> struct svc_req * rqstp, <br/> Register svcxprt * Transp) <br/> # endif <br/> {<br/> Union {<br/> square_in squareproc_2_arg; <br/>} argument; <br/> Union {<br/> square_out squareproc_2_res; <br/>}result; <br/> bool_t retval; <br/> xdrproc_t _ xdr_argument, _ xdr_result; <br/> bool_t (* local) (char *, void *, struct Svc_req *); </P> <p> switch (rqstp-> rq_proc) {<br/> case nullproc: <br/> (void) svc_sendreply (Transp, (xdrproc_t) xdr_void, (char *) null); <br/> return; </P> <p> case squareproc: <br/> _ xdr_argument = (xdrproc_t) xdr_square_in; <br/> _ xdr_result = (xdrproc_t) xdr_square_out; <br/> Local = (bool_t (*) (char *, void *, struct svc_req *) squareproc_2_svc; <br/> break; </P> <p> default: <br/> svcerr_noproc (Transp); <Br/> return; <br/>}< br/> memset (char *) & argument, 0, sizeof (argument); <br/> If (! Svc_getargs (Transp, (xdrproc_t) _ xdr_argument, (caddr_t) & argument) {<br/> svcerr_decode (Transp); <br/> return; <br/>}< br/>/* unlock RPC call thread */<br/> # ifdef using_mt_rpc_with_lock <br/> If (pmutex) pthread_mutex_unlock (pmutex ); <br/> # endif <br/>/* Call user define function */<br/> retval = (bool_t) (* local) (char *) & argument, (void *) & result, rqstp); <br/> If (retval> 0 &&! Svc_sendreply (Transp, (xdrproc_t) _ xdr_result, (char *) & result) {<br/> svcerr_systemerr (Transp); <br/>}< br/> If (! Svc_freeargs (Transp, (xdrproc_t) _ xdr_argument, (caddr_t) & argument) {<br/> fprintf (stderr, "% s", "unable to free arguments "); <br/> exit (1); <br/>}< br/> If (! Square_prog_2_freeresult (Transp, _ xdr_result, (caddr_t) & result) <br/> fprintf (stderr, "% s", "unable to free results "); </P> <p> return; <br/>}</P> <p> int <br/> main (INT argc, char ** argv) <br/> {<br/> Register svcxprt * Transp; <br/> # ifdef using_mt_rpc_with_lock <br/>/* Create a mutex object for RPC calling */<br/> pthread_mutex_init (& rpcmutex, null ); <br/> # endif <br/> pmap_unset (square_prog, square_v ERS); </P> <p> Transp = svcudp_create (rpc_anysock); <br/> If (Transp = NULL) {<br/> fprintf (stderr, "% s", "cannot create UDP Service. "); <br/> exit (1); <br/>}< br/> If (! Svc_register (Transp, square_prog, square_vers, protocol, ipproto_udp) {<br/> fprintf (stderr, "% s", "unable to register (square_prog, square_vers, UDP ). "); <br/> exit (1); <br/>}</P> <p> Transp = svctcp_create (rpc_anysock, 0, 0 ); <br/> If (Transp = NULL) {<br/> fprintf (stderr, "% s", "cannot create TCP Service. "); <br/> exit (1); <br/>}< br/> If (! Svc_register (Transp, square_prog, square_vers, protocol, ipproto_tcp) {<br/> fprintf (stderr, "% s", "unable to register (square_prog, square_vers, TCP ). "); <br/> exit (1); <br/>}</P> <p> svc_run (); <br/> fprintf (stderr, "% s", "svc_run returned"); <br/> exit (1); <br/>/* notreached */<br/>}< br/>

At line #70 of the code, we can see that after creating a thread for the customer, after the thread is created, wait for the lock in line #73, that is, wait for the parameter to be parsed by RPC and wait until the server is called to process the code. In row #152, I released the lock so that the lock can be returned in the main thread, and the auxiliary thread will enter the custom server processing function. In this way, a reasonable wait is achieved.

The program code and MAKEFILE file will be sent here. If you need such information, you can test it and come up with more ideas.

Square. x

Struct square_in {<br/> longarg1; <br/>}; </P> <p> struct square_out {<br/> longres1; <br/> }; </P> <p> program square_prog {<br/> Version square_vers {<br/> square_outsquareproc (square_in) = 1; <br/>/* procedure number = 1 */<br/>}= 2;/* version number = 2 */<br/>}= 0x31230000; /* program number = 0x31230000 */<br/>

Server. C service user Processing Function Location

# Include <unistd. h> <br/> # include <pthread. h> </P> <p> # include "Square. H "</P> <p> bool_t <br/> squareproc_2_svc (square_in * indium, square_out * outp, struct svc_req * rqstp) <br/>{< br/> printf ("thread % LD started, Arg = % LD/N", <br/> pthread_self (), Indium-> arg1 ); <br/> sleep (5); <br/> outp-> RES1 = Indium-> arg1 * Indium-> arg1; <br/> printf ("thread % LD done/N", pthread_self (); <br/> return (true ); <br/>}</P> <p> int <br/> square_prog_2_freeresult (svcxprt * transp, xdrproc_t xdr_result, <br/> caddr_t result) <br/>{< br/> xdr_free (xdr_result, result); <br/> return (1); <br/>}< br/>

Client. the C client calls the clnt_destroy function, especially when it is not called during an error, A connection is maintained between the client and the server (a file is opened in Linux, many TCP connections (Files) will be opened on the server, which will lead to errors in the server program due to the number of files opened by the operating system process (when the file is opened ). So be sure to pay attention to it!

# Include <rpc/rpc. h> </P> <p> # include "Square. H "</P> <p> int <br/> main (INT argc, char ** argv) <br/> {<br/> client * Cl; <br/> square_inin; <br/> square_outout; </P> <p> If (argc! = 3) {<br/> printf ("Usage: client <postname> <integer-value>"); <br/> exit (0 ); <br/>}</P> <p> Cl = clnt_create (argv [1], square_prog, square_vers, "TCP"); </P> <p> in. arg1 = atol (argv [2]); <br/> If (squareproc_2 (& in, & out, Cl )! = Rpc_success) {<br/> printf ("% s", clnt_sperror (CL, argv [1]); <br/> clnt_destroy (CL ); <br/> exit (0); <br/>}< br/> clnt_destroy (CL); <br/> printf ("Result: % LD/N", out. RES1); <br/> exit (0); <br/>}< br/>

Makefile

Progs = client server <br/> # cancele when you don't need thread lock to use mulitthread <br/> macro =-dusing_mt_rpc_with_lock <br/> cflags =-g-DH- wall-ddebug $ (macro) <br/> # cflags =-wall <br/> linkflags =-lpthread <br/> cc = GCC </P> <p> All: $ {progs} </P> <p> Square. h square_clnt.c square_svc.c square_xdr.c: Square. x <br/> rpcgen-C-m square. x </P> <p> square_clnt.o: square_clnt.c square. h </P> <p> square_svc.o: square_svc.c square. h </P> <p> client: Square. h client. O square_clnt.o square_xdr.o <br/>$ {CC }$ {cflags}-o $ @ client. O square_clnt.o square_xdr.o </P> <p> server: Square. h server. O square_svc.o square_xdr.o <br/>$ {CC }$ {cflags}-o $ @ server. O square_svc.o square_xdr.o $ (linkflags) </P> <p> clean: <br/> RM-f * o <br/>

 

Compile the program by using the make command, and then overwrite the square_svc.c file before compiling to implement multithreading.

Finally, I found that if the client is terminated too early, the server will fail. The problem lies in line #156 of the square_svc.c file. The problem occurs when the TCP reply is received:

Program received signal SIGSEGV, segmentation fault.
[Switching to thread-1208173680 (lwp 28672)]
0x00ac7ef5 in writetcp () from/lib/libc. so.6
(GDB) Where
#0 0x00ac7ef5 in writetcp () from/lib/libc. so.6
#1 0x00aca570 in xdrrec_endofrecord_internal () from/lib/libc. so.6
#2 0x00ac7e00 in svctcp_reply () from/lib/libc. so.6
#3 0x00ac682c in svc_sendreply_internal () from/lib/libc. so.6
#4 0x08048b39 in square_prog_2_thread_func (rqstp = 0xbfe9af18, Transp = 0x8f36708, pmutex = 0x80491a4)
At square_svc.c: 156
#5 0x08048a3a in square_prog_thread_func (Param = 0x8f33008) at square_svc.c: 99
#6 0x00b462db in start_thread () from/lib/libpthread. so.0
#7 0x00aa014e in clone () from/lib/libc. so.6
(GDB)
This occurs when the thread prepares to enable the returned results for the server user function call. The reason should not be a segment error.

If you want the server to work stably, you need to handle this error. Hope you can solve the problem.