200 points for optimization recommendations, how to solve

200 Points for optimization recommendations
The problem is this:

C, A-B

A is our partner and they are responsible for synchronizing the data to us via the HTTP protocol.
B is we are responsible for receiving a push over the data, do business logic processing, and then put the collated data to our partner C (there are many C).


The overall architecture of my program is this:
1 There is a portal file, is actually a PHP file, is responsible for receiving a data, stored in the database.
2 I call PHP CLI mode with the Nohup command to open a PHP process to run a PHP file in the system background processing of the following things: from the database to fetch 10 data at a time, loop out each one, and then according to the business logic, according to each data different, forwarded to a different C ( is actually to access different URLs).


A For me to synchronize the data, a second may come 10, they are multi-threaded packet. And my side through the storage, one time to remove 10 data, the loop push to a different URL address, after receiving their interface to my response to the ' OK ', mark, and then push the next piece of data, if I set the timeout time 5 seconds after not received the other's ' OK ', then the re-send, the timeout is still 5 seconds, A total of 2 re-hair. is equal to a data up to push 3 times, 3 times can not push past the words, no matter. Also, I set sleep for 1 seconds between each push data.


Problems encountered:
Normally, push a piece of data should be within 2 seconds, but if a C, his interface by the network cause or the problem of the program, I 3 times the push is not pushed past (that is, his interface response speed of more than 5 seconds or not responding at all), it will make my data queue up to 18 seconds slack. This will make the push speed much lower! I looked at it today and my data queue was about one hours slower than the normal speed. Is there any good way to solve this problem?
PHP itself does not support multi-threading, I can not set different threads to send different C push data. So a C interface responds slowly, affecting all C, affecting the entire data queue.
I do not know if any friends have done this aspect of things, teach point of experience. In addition to request C to improve the interface response speed, and ensure that the network unblocked. Is there a way to solve the problem on my own side? Some of my partners they use Java, can establish thread pool multithreading, but PHP this is more helpless.


In fact, in the past, I used the way is synchronous forwarding, that is, after receiving a data, immediately processing business logic, and then immediately forward data synchronization to C, not into the queue. This is not the case, but it will create a lot of other problems, so I now change to the way this queue.

I've thought about two solutions that don't know if it works:
1 PHP itself does not support multi-threading, but Linux under the Curl,wget, or Apache band AB. They support multithreading, can I use PHP to invoke them to implement multi-threaded push data? Assign a thread to each of the different C, so that a C block will not affect the other C.
2 does not support multi-threading, I use multi-process, I give each different C open a separate PHP process, so also can avoid the interaction between C. But I'm not sure how much it will affect the operating system if you run so many processes with infinite cycles. At present, we have a business of about 10 c.


I hope you give some advice, the guidance, thank you!

------Solution--------------------
Open a multi-process, give each c a separate PHP process,
Impact, on the same time Open 2, has been used, a monitoring some other processes, a file operation, all die Cycle sample run,
It is not found that the dead loop itself has an impact on the load. When scheduling file operations, it will certainly affect the load. cpu/hard Drive Read and write strong enough, should be no problem, or reduce to 5, each corresponding to 2 C

------Solution--------------------
Frequently check the library do not know if there is a bottleneck, if it is a direct data, not for processing, as well as in addition to the repository, and put in memory, generate queues, let the PHP process go to the memory queue to take
------Solution--------------------
Resolving with soap
------Solution--------------------
Using SOAP to solve high availability
The principle is that soap listens to receive data
Then soap background starts multi-process

------Solution--------------------
By the right, not waiting to return
The x.php runs continuously, periodically obtains data from the memory queue and marks the data in memory, then pushes the C-side after processing, but does not wait for the return
x.php get data, only untagged data for a period of time
The C-end returns the request, sends to another, m.php, receives the return, and compares with the memory queue, deletes the compliant
------Solution--------------------
PHP CLI mode can be multithreaded but weak under non-win

HTTP is a stateless protocol that can be done in many processes, but there's a serious problem.

A-> 10+/s
->c 10/5s
That is, C is sent at a speed lower than the receive speed of a to cause the previous C has not exited before the start of a C finally stacked N process system crashes or each C process caused a large data delay

Each C can be given a single process, but each time not limited to 10 should be a backlog of a few on the number of such effective control process quantity and delay the maximum delay is basically the run time + transmission time
or each c separate process + multi-threaded queue each thread 10 backlog more than 10 to start a new thread basically each process starts 20 threads enough to guarantee synchronization

Data provide my practice is passive access after all, the user does not have a few more than a few K-node XML generation send load is not high and can take advantage of the cache

Oh, soap may be more appropriate to support upstream and downstream
------Solution--------------------
My advice:

1, put your current processing b->c PHP CLI program under Apache, that is, http://www.x.com/b2c.php

2, open a separate PHP CLI process php.exe db_read.php, to cycle through the database and read the field to submit to b2c.php
You can submit 10 at a time, no problem, because Apache supports multithreading:)


------Solution--------------------
I don't know much about this, but I saw a "server push" technology a while ago. may be useful.

In addition, I say SQL query this piece, assuming that your SQL database does not have much optimization, each query 10, so speed will have bottlenecks, of course, perhaps the requirements of the data for this query can be fully satisfied.

With your push speed to reason, we recommend one-time reading data such as 100, staged as a file, file operation is faster than the database operation.

Push out 10 Delete 10, when only 10 left, then read 100. This can reduce the number of times the database is read.

To optimize, you need to optimize to the teeth. Other aspects I do not think of any way, wood sparse study shallow, can not help.
------Solution--------------------
Open a multi-process, give each c a separate PHP process,
Impact, on the same time Open 2, has been used, a monitoring some other processes, a file operation, all die Cycle sample run,
------Solution--------------------
cite the landlord Shadowsniper post:
The problem is this:

C, A-B

A is our partner and they are responsible for synchronizing the data to us via the HTTP protocol.
B is we are responsible for receiving a push over the data, do business logic processing, and then put the collated data to our partner C (there are many C).


The overall architecture of my program is this:
1 There is a portal file, is actually a PHP file, is responsible for receiving a data, stored in the database.

