Python crawler engineer interview summary, python Crawler
Note: answers are generally found online.
1. Statement of Understanding if _ name _ = 'main'
2. How does python perform memory management?
3. Write a piece of Python code to delete duplicate elements in a list.
4. How to copy an object in Python? (Assignment, shortest copy, and deep copy)
5. What are the usage and functions of ipvt?
6. Differences between the _ new _ and _ init _ methods in Python
7. Common network data crawling Methods
8. Anti-crawler policies and solutions
9. Differences between urllib and urllib2
10. Design a crawler Scheme Based on session login verification
11. List network data packets used by web crawlers and parse packets
12. Familiar crawler frameworks
13. Python server deployment process and environment isolation
14. What are the similarities and differences between Django and Flask?
15. Write a singleton mode in Python
16. Linux service deployment script commands (including shell scripts for starting and stopping)
17. Have you used multithreading and Asynchronization? In addition, what methods have you used to improve crawler efficiency?
18. Differences between POST and GET
1) Statement of Understanding if _ name _ = 'main'
_ Name _ is the current Module name. When the module is directly run, the module name is _ main __, that is, the current module. When the module is imported, the module name is not _ main __, that is, the code will not be executed.
2) How does python perform memory management?
A. Object reference counting mechanism
Python uses reference count internally to keep track of objects in memory. Python records the number of references of objects, that is, reference count. when an object is created, a reference count is created, when the object is no longer needed and its reference count is 0, it is reclaimed.
B. Garbage Collection
1> when the reference count of an object is zero, it will be disposed of by the garbage collection mechanism.
2> when two objects a and B reference each other, the del statement can reduce the reference count of a and B and destroy the name used to reference the underlying object. However, because each object contains an application for other objects, the reference count is not zero and the object is not destroyed. (Resulting in Memory leakage ). To solve this problem, the interpreter periodically executes a loop detector to search for loops of inaccessible objects and delete them.
C. Memory Pool Mechanism
Python provides a garbage collection mechanism for memory, but it puts unused memory into the memory pool instead of returning it to the operating system.
1> Pymalloc mechanism. To accelerate the execution efficiency of Python, Python introduces a memory pool mechanism to manage the application and release of small memory blocks.
2> in Python, all objects smaller than 256 bytes use the Allocator implemented by pymalloc, while large objects use the system's malloc.
3> Python objects, such as integers, floating-point numbers, and lists, all have their own private memory pools, and objects do not share their memory pools. That is to say, if you allocate and release a large number of integers, the memory used to cache these integers cannot be distributed to floating point numbers.
3) write a piece of Python code to delete duplicate elements in a list.
#1. use the set function list = [1, 3, 4, 5, 51, 2, 3] set (list) #2. use dictionary functions, >>> a = [1, 2, 4, 2, 4, 5, 6, 5, 7, 8, 9, 0] >>> B ={}>> B = B. fromkeys (a) >>> c = list (B. keys ()> c
4) how to copy an object in Python? (Assignment, shortest copy, and deep copy)
A value (=) is a new reference of an object. modifying any variable affects the other.
Shallow copy: Creates a new object, but it contains references to the items contained in the original object (if one object is modified by reference, the other will also be changed) {1, full slicing method; 2, factory functions, such as list (); 3, copy () function of the copy module}
Deep copy: Creates a new object and Recursively copies the objects it contains (modifying one of them will not change) {copy module's deep. deepcopy () function}
5) What are the usage and functions of ipvt?
Try... Else T... Else T... Else...
Execute the statement in try. If an exception is thrown, the execution process jumps to the limit t statement. Execute the command in sequence for each branch. If the exception is matched with the exception group in branch T, execute the corresponding statement. If none of the except T matches, the exception will be passed to the top-level try code that calls this code next.
If the statement in try is executed normally, the else block code is executed. If an exception occurs, the finally statement is not executed.
6) differences between the _ new _ and _ init _ methods in Python
_ New __: called when an object is created. an instance of the current object is returned. You can use _ new _ to implement a singleton.
_ Init __: it is called after an object is created. It initializes some instances of the current object and has no return value.
7) Common network data crawling Methods
- Regular Expression
- Beautiful Soup
- Lxml
8) have encountered anti-crawler policies and solutions
1. Anti-crawler through headers
2. user behavior-based crawling: (The Access Frequency of the same IP address within a short period of time)
3. Dynamic Web page anti-crawler (data request through ajax or JavaScript)
4. Encrypt part of the data (garbled data)
Solution:
You can customize headers for basic webpage capturing and add headers data.
Multiple proxy ip addresses are used to capture or set the capture frequency to a lower level. Dynamic Web pages can be captured using selenium + phantomjs.
Selenium can be used to encrypt part of the data and identify it using python's pytesseract library. However, the most direct and slow method is to find the encryption method for reverse reasoning.
9) differences between urllib and urllib2
Both urllib and urllib2 accept URL Request modules, but urllib2 can accept a Request class instance to set URL Request headers. urllib can only accept URLs. Urllib cannot disguise your User-Agent string.
Urllib provides the urlencode () method to GET the generation of query strings, but urllib2 does not. This is why urllib is often used with urllib2.
10) Design a crawler Scheme Based on session login verification
11) list network data packets used by web crawlers and parse packets
Network Packet urllib, urllib2, requests
Parsing package re, xpath, beautiful soup, lxml
12) familiar crawler frameworks
The Scrapy framework will answer your questions based on your actual situation.
13) Python server deployment process and environment isolation
14) What are the similarities and differences between Django and Flask?
15) Write a singleton mode in Python
class Singleton(object):_instance = Nonedef __new__(cls, *args, **kw):if not cls._instance:cls._instance = super(Singleton, cls).__new__(cls, *args, **kw) return cls._instanceclass MyClass(Singleton): a = 1one = MyClass()two = MyClass()id(one) = id(two)>>> True
16) service deployment script commands in Linux (including shell scripts for starting and stopping)
17) Have you used multithreading and Asynchronization? In addition, what methods have you used to improve crawler efficiency?
- Scrapy-redis distributed crawling
- For targeted crawling, replace xpath with regular expressions.
18) Difference between POST and GET
- Low Security of GET data transmission and high security of POST data transmission because parameters are not stored in browser history or web server logs;
- We recommend that you use the GET method when querying data. We recommend that you use the POST method when adding, modifying, or deleting data;
- GET transmits data in the url, and the data information is placed in the Request Header; the POST request information is placed in the Request body to transmit data;
- GET transfers a small amount of data, and can only send data in the Request Header, while POST transfers a large amount of data information, generally not limited;
- In terms of execution efficiency, GET is better than POST.
19) What is a lambda function? What are its advantages?
Lambda expressions are usually used when a function is needed, but you don't want to bother naming a function, that is, anonymous functions.
Lambda functions: the primary purpose is to give pointers to short callback functions.
lambda [arguments]:expression>>> a=lambdax,y:x+y>>> a(3,11)