Python crawler Questions

1. What is the difference between TCP and UDP? TCP (transmission Control Protocol, transmission Protocol) is a connection-oriented protocol, which means that a reliable connection must be established with each other before sending or receiving data. A TCP connection has to go through three "dialogs" to build up, and the process is very complex and simply describes the simple process of three conversations: Host A sends a connection request packet to Host B: "I want to send you data, OK?" "This is the first conversation; Host B sends a consent connection to host A and requires synchronization (synchronization is the two host one in the send, one in the receiving, coordination work) packet:" Can, when do you send? " "This is the second conversation; Host a then sends a packet to confirm that Host B's requirements are synchronized:" I'll send it now, you go on! " ", this is the third dialogue.　　Three times the purpose of the "conversation" is to synchronize the sending and receiving of packets, and after three "conversations", host a formally sends the data to Host B.　　TCP three-time handshake process: 1 Host A to host B by sending a data segment with the flag bit of the synchronization sequence number to Host B, to host B request a connection, through this data segment, host a tells Host B two things: I want to communicate with you, you can use which serial number as the starting data segment to respond to me. 2 when Host B receives a request from host A, it responds to host A with a data segment with an acknowledgment answer (ACK) and a synchronous sequence number (SYN) flag bit, and also tells host a two things: I have received your request, you can transfer the data; which pins XLR serial number you want to use as the starting data segment to respond to my 3　　When host a receives this data segment, it sends a confirmation reply confirming that it has received the data segment of Host B: "I have received a reply, I am now going to start transmitting the actual data so that 3 times the handshake is complete, and host A and Host B can transfer the data." 3-Time handshake characteristics: No application layer of Data SYN this flag bit only when the TCP build connection will be placed 1 handshake completion after the SYN flag bit is set 0 TCP to make a connection to 3 handshake, and the disconnection to be carried out 4 times: 1 When host a completes the data transfer, the control bit fin 1, proposed to stop TCP　　Connection Request 2 Host B responds to the fin after receiving it, confirming that the side upward TCP connection will be turned off, the ACK will be set to 1 3 by the B end of the reverse direction of the shutdown request, the Fin 1 4 host A to Host B to confirm the request, will be ACK 1, two-direction closure end.　　It can be seen from the three handshake and four disconnects of TCP that TCP uses the connection-oriented communication method, which greatly improves the reliability of data communication, so that the transmitting data end and the receiving end have interaction before the formal transmission of the date, which lays a reliable foundation for the formal transmission of data. Noun explanation: The control bit of the ACK TCP headerFirst, confirm the data. Confirm that the data segment was sent to the destination before it was used to tell the sender that the serial number was received. For example, the confirmation number is x, that is, the first X-1 data segment received, only when ack=1, the confirmation number is valid, when ack=0, the confirmation number is invalid, this will require retransmission of data,　　Ensure the integrity of the data. 　　SYN Synchronous serial number, TCP establishes the connection will be this position 1 FIN send end to complete the sending task bit, when the TCP complete data transfer needs to disconnect, the proposed disconnection party will this position 1 TCP header structure: Source port 16-bit target port 16-bit serial number 32-bit response sequence 32-bit　　TCP Header length 4-bit reserved 6-bit control Code 6-bit window size 16-bit offset 16-bit check and 16-bit option 32-bit (optional) so we get the minimum length of the TCP header, which is 20 bytes. UDP (User data Protocol, Subscriber Datagram Protocol) (1) UDP is a non-connected protocol that transmits data before the source and terminal do not establish a connection, and when it wants to transmit it simply crawls data from the application and throws it to the network as quickly as possible.　　On the sending side, UDP transmits data at a speed that is limited by the speed of the application-generated data, the ability of the computer, and the transmission bandwidth; At the receiving end, UDP places each message segment in the queue and the application reads a message segment from the queue each time.　　(2) Because the transfer data does not establish a connection, so there is no need to maintain the connection state, including the sending and receiving status, so a server can simultaneously transmit the same message to multiple clients.　　(3) The header of the UDP packet is very short, with only 8 bytes, and the extra overhead of 20 byte packets relative to TCP is small.　　(4) Throughput is not regulated by congestion control algorithms, only the rate of data generated by the application software, the transmission bandwidth, the source side and the performance limit of the terminal host.　　(5) UDP uses the best effort to deliver, that is, does not guarantee reliable delivery, so the host does not need to maintain a complex link state table (there are many parameters). (6) UDP is message-oriented. The sender's UDP message to the application is delivered down to the IP layer after the header is added.　　Instead of splitting and merging, the boundaries of these messages are preserved, so the application needs to select the appropriate message size.　　We often use the "ping" command to test the TCP/IP communication between the two hosts is normal, in fact, the principle of "ping" command is to send UDP packets to the other host, and then the other host to confirm the receipt of the packet, if the packet arrives timely feedback back, then the network is through.　　UDP header structure: Source port 16-bit destination port 16-bit length 16-bit checksum 16-bitSummary the difference between TCP and UDP: 1. Based on connection and no connection, 2. System Resource requirements (TCP more, UDP less), 3.UDP program structure is simple, 4. Flow mode and datagram mode, 5.TCP to ensure data correctness, UDP may drop packets, TCP guaranteed Data order, UDP is not guaranteed.　　2. Knowledge of in-memory stacks and heaps? Stack: What is a stack, which is a special area of your computer's memory that is used to store temporary variables created by each function (including the Mian () method). Stack is filo, is advanced after the principle of the structure, it is closely managed by the CPU and fully utilized. Each time the function declares a new variable, it is "pushed" into the stack. Then each time a function exits, all variables defined in the function are freed (in other words, deleted).       Once the variables in the stack are released, the area becomes available and provided to the variables in the other stacks. The advantage of storing variables with stacks is that the memory is managed by you. You don't have to create the memory manually, and you don't have to manually free the memory when you don't need it. In addition, CPU stack memory is very efficient.      Read-out and write-stack variables are fast. The key to understanding the stack is understanding the concept, when a function exits, all its variables are popped out of the stack and will disappear forever. Therefore, the nature of the variables in the stack is local. This is related to what we originally understood as variable scopes or local or global variables.     In C, a common bug is to try to access a function in the stack from a function outside of your program (after the function has exited). Another feature of the stack we should remember is that there is a limit to the size of the variables stored in the stack.     Create variables on the heap without consideration.      Summary stack: A, the extension and expansion of the stack is the function of pressure or the introduction of local variables.      b, we do not have to manage the memory ourselves, variable creation and release are automatic.　　c, the variables in the stack exist only when the function creation runs. Heap: The heap is also an area of our computer's memory, but he is not automatically managed. And it's not being managed closely by the CPU. It is a more free area of memory (very large). To create memory on the heap, we have to use malloc () or calloc (), which are compiled in C language. Once you allocate memory on the heap, you must use free () to destroy it when you are not in need. If you do not destroy or destroy the failure, your program will have a memory leak.     In other words, the heap memory is always there and other processes are unusable. Unlike stacks, heaps have no limit on the size of variables (except for the physical limitations of your computer). Heap memory read-out andWrites are slow because it must use a pointer graph to access heap memory.          Stack and heap advantages and disadvantages: stack: A, quick access.          b, there is no need to explicitly create categorical variables, because it is automatically managed.          C, the space is effectively managed by the CPU, the memory will not become fragmented.        D, only the local variable e, limited by the stack size (depending on the operating system) F, the variable cannot be resized.          Heap: A, variable can be global access B, no memory size limit C, (relative) Access slower d, no efficient use of space, with block memory creation and destruction, memory may become fragmented. E, you must manage memory (the creation and destruction of variables you have to be responsible for) F, variable size can be adjusted with ReAlloc () 3, the crawler framework scrapy workflow A, spider parsing the response down the download, return to item or links B, ite M or link after spidermiddleware process_spider_out () method, to engine C, engine will item to item pipeline, links to Scheduler D, in the scheduler, The requests object is generated using the scrapy built-in fingerprint function to generate a fingerprint object E, if the requests object in the "T" filter parameter is set to False, and the thumbprint of the requests object is not in the information fingerprint queue, Then put the request object in the priority queue F, get the request object from the priority queue, to engine G, engine will send the request object to the download, during the Downloadmiddleware Process_ The request method H, the downloader completes the download, obtains the response object, passes the object to the engine, during the Downloadmiddleware process_response () method I,　　　　Engine will get the response object to the spider to parse, during the spidermiddleware of the Process_spider_input () method J, starting from a Loop 4, dictionary, list query when the complexity of the time? A list is a sequence that can be understood as an array in a data structure, the dictionary can be understood as a hashmap,python in the data structure of the list object's storage structure using a linear table, so its query complexity is O (n), and the storage structure of the Dict object is a hash list (hash table), which in the optimal case of the query complexity O (1). Dict memory is slightly larger than the list, it will be about 1.5 times times.　　5. What happens if there is no termination condition in recursion? Concept: Recursive algorithm is a direct or indirect process of invoking its own algorithm.　　In computer programming, recursive algorithm is very effective to solve a large class of problems.　　Feature: ① recursion is the invocation of itself in a procedure or function.　　② when using a recursive strategy, there must be a definite recursive condition called a recursive exit. ③ recursive algorithm is usually very concise, but the recursive algorithm is less efficient in solving problems.　　Therefore, we generally do not advocate the use of recursive algorithm design program. ④ in the process of recursive invocation, each layer of the system's return points, local variables, etc. open up a stack to store. Too many recursive functions tend to cause stack overflow and so on. Therefore, the recursive algorithm is generally not advocated to design the program recursion if there is no termination condition will cause the recursive call to become a dead loop and not end normally, and will cause stack overflow written 1, remove duplicate elements in the list copy code def distinct_list (list): d_l = [] for I In List:if I not in d_l:d_l.append (i) Return d_l copy Code 2, get a sub-list of the list that satisfies the list of elements that are contiguous in the original list, and the sum of all the elements of the child list Biggest. For example [1,-2, 3,-1, 2] = = [3,-1, 2]

　　

Python crawler Questions