200 The problem with optimization suggestions is as follows: a-& gt; B-& gt; ca is our partner who is responsible for synchronizing data to us through http protocol. B is us. we are responsible for receiving the data pushed by a, processing the business logic, and then pushing the sorted data to our partner c (there are many c ). The overall architecture of my program is as follows: 1. there is an entry file, which is actually a php file, which is responsible for receiving a's data and saving it to the database. 2. I used the nohup command to call php 200 for optimization suggestions.
The problem is as follows:
A-> B-> c
A is our partner who is responsible for synchronizing data to us through the http protocol.
B is us. we are responsible for receiving the data pushed by a, processing the business logic, and then pushing the sorted data to our partner c (there are many c ).
The overall architecture of my program is as follows:
1. there is an entry file, which is actually a php file, which is responsible for receiving a's data and storing it in the database.
2. I used the nohup command to call the php cli mode to open a php process and run a php file to process the following things in the system background: retrieve 10 pieces of data from the database at a time, and execute every fetch in a loop, then, based on the business logic and different data entries, forward the data to different c (actually, access different URLs ).
When a synchronizes data to me, there may be more than 10 records in one second. they are multi-threaded packages. On the other hand, I read 10 pieces of data into the database at a time, push them cyclically to different url addresses, and mark them after receiving the 'OK' from their interface, next, push the next data. if I haven't received the other party's 'OK' after setting the time-out period of 5 seconds, resend the data. the time-out period is still 5 seconds, and resend the data twice in total. It means that a piece of data can be pushed three times at most, and if it cannot be pushed three times, it will not matter. In addition, I set sleep for 1 second between each push data.
Problems:
Under normal circumstances, pushing a piece of data should be less than 2 seconds, but if a c, his interface is subject to network reasons or program problems, I haven't pushed the push for three times (that is to say, the response speed of his interface is greater than 5 seconds or there is no response at all), it will make my data queue idle for a maximum of 18 seconds. This will greatly reduce the push speed! Today, I saw that my data queue is about an hour slower than normal. Is there any good way to solve this problem?
Php itself does not support multithreading. I cannot set different threads to push data to different c threads. Therefore, the slow response of a c interface affects all c and the entire data queue.
I wonder if you have any friends who have done this. In addition to requiring c to increase the interface response speed and ensure smooth network access. Is there a solution on my own side? Some of my partners use java to create thread pools for multi-thread packet sending, but php is helpless.
In fact, in the past, I used synchronous forwarding, that is, after receiving data from a, I immediately processed the business logic and immediately forwarded the data to c without entering the queue. In this way, there are no such problems, but there will be many other problems, so I am changing to this queue method now.
I thought about two solutions and I wonder if they are feasible:
1 php itself does not support multithreading, but curl, wget, or AB in apache in linux. They support multithreading. Can I use php to call them to push data in multiple threads? Assign a thread to each different c. in this way, a c block will not affect other c.
2. multithreading is not supported. I use multi-process. I open a separate php process for each different c, which can avoid the mutual influence between c. But I don't know how much impact the operating system will be caused by the infinite endless loop of so many processes. At present, we have about 10 c businesses.
I hope you can give me some advice. thank you!
------ Solution --------------------
Enable multi-process and separate a php process for each c,
The impact is that two processes are enabled at the same time. one is used to monitor some other processes, and the other is used to perform file operations,
No effect is found on the load. When a file is scheduled, the load is definitely affected. Cpu/hard disk read/write is strong enough, it should be OK, or reduce to 5, each corresponding to 2 c
------ Solution --------------------
Frequent database check does not know whether there is a bottleneck. if direct data from a is not processed, it is better to save the database and put it in the memory to generate a queue so that the php process can get it in the memory queue.
------ Solution --------------------
Solve with SOAP
------ Solution --------------------
High feasibility of using SOAP
The principle is that the soap listener receives data.
Then, the soap background starts multiple processes.
------ Solution --------------------
By the way, do not wait for the return
X. php is always running. it regularly obtains data from the memory queue, marks the data in the memory, and pushes the data to the c end after processing, but does not wait for the response
X. php only obtains unlabeled data within a period of time
C-end return request, sent to another, m. php, receives the response, and compares it with the memory queue to delete the matching
------ Solution --------------------
The php cli mode can be multi-threaded but weak in non-WIN scenarios.
HTTP is a stateless protocol that can be fully implemented by multiple processes, but there is a serious problem.
A-> 10 +/S
-> C 10/5 S
That is to say, the sending speed of C is lower than the receiving speed of A, which causes the previous C to stop and start another C. finally, N processes are Accumulated. the system crashes, or each C process causes A large data delay.
Each C can be assigned a single process, but each time it is not limited to 10 items, it should be a backlog of several items to send a few so as to effectively control the number of processes and the maximum latency is basically the running time + transmission Time
Or every C single process + multi-thread gets a line-up of 10 threads with a backlog of more than 10 to start a new thread. Basically, every process starts 20 threads to ensure synchronization.
The method of data provision is passive access. after all, the XML generation and sending load of a few K nodes is not high and the cache can be used.
Haha, SOAP may be more suitable, but it must be supported by both upstream and downstream.
------ Solution --------------------
My suggestions:
1. put your current php cli program for processing B-> C under apache, that is, http://www.x.com/ B2C .php
2. open an independent php cliinput php.exe db_read.php to read the database cyclically and submit the read fields to B2C. php.
You can submit 10 at a time, because apache supports multiple threads :)
------ Solution --------------------
I don't know much about this, but I saw a "server push" technology some time ago. It may be useful.
In addition, let me talk about SQL queries. if your SQL database is not optimized too much, there will be 10 queries each time, which will lead to a bottleneck. of course, perhaps the data requirements can fully meet this query.
We recommend that you read 100 data records at a time and save them as a file. after all, file operations are faster than database operations.
Push 10 to delete 10. when only 10 are left, read 100 more. This reduces the number of database reads.
To optimize, you need to optimize the teeth. In other aspects, I don't think of any way. I am not very familiar with materials and can't help.
------ Solution --------------------
Enable multi-process and separate a php process for each c,
The impact is that two processes are enabled at the same time. one is used to monitor some other processes, and the other is used to perform file operations,
------ Solution --------------------
Reference the post of ShadowSniper:
The problem is as follows:
A-> B-> c
A is our partner who is responsible for synchronizing data to us through the http protocol.
B is us. we are responsible for receiving the data pushed by a, processing the business logic, and then pushing the sorted data to our partner c (there are many c ).
The overall architecture of my program is as follows:
1. there is an entry file, which is actually a php file, which is responsible for receiving a's data and storing it in the database.
