Six experiences of using Python in DropBox

The following is an example of how DropBox uses Python:

1.1 Use Python

99.9% of Dropbox code is written in Python. Python uses server-side, Desktop client, website control logic, back-end API and Analysis

On the Android platform, Python cannot be used due to memory restrictions.

Run a single code in Python. Use tools such as PyObjs, WxPython, types, py2exe, py2ap, and PyWin32 to run Dropbox on Windows, Mac, and Linux systems.

Advantages:

Developers communicate and express ideas through Python

Easy to learn, easy to read, easy to write, easy to get started with new people

Disadvantages:

Don't be silly

It consumes too much memory and runs too slowly. It is not a big problem on the server side, as long as you buy a large machine. However, on the client side, do not expect old Power PC users to upgrade their devices.

Mixed Programming of Python and C may cause problems, it's hard to figure out what you want to do with fixed memory and CPU (Coding in a mixed environment of Python and C creates problems because it's hard to profile your SS the language boundaries like you want to do when fixing memory and CPU problems .)

Memory fragmentation is a good idea that the script language may not be a long-term running process.

1.2 let it work from the very beginning(Just Work Baby)

If no matter what file system you are, What Operating System, what applications you are using. This product should always work only.

On a wide range of supported platforms, you can analyze various errors encountered. Python helps them quickly iterate.

1.3 early release(Release Early)

It is easy to code and publish it within one day using Python

1.4 It is easy to optimize the CPU by using C in a loop(Use C for Inner Loops, Optimizing CPU is easy)

Is a way to solve the problem of slow speed.

Optimize internal cycles to reduce CPU time.

Loop in Python consumes 44% more time than in C.

Python Virtual Machine bytecode scheduling is slow.

There are many tools to tune the CPU.

CPU optimization is usually limited to small code segments

1.5 Round Robin is not easy to expand(Polling 30 million clients all over the world doesn't scale)

Create an HTTP notification structure to prevent clients from polling the server.

1.6 It is difficult to optimize the memory by customizing the memory distributor.

There was one of the biggest problems for a while. A large amount of memory that can be used, but the memory will never be released. For large-scale synchronization, they can use up to 1.5 GB, And now they rarely use more than 100 MB.

It is difficult because:

Few tools are available to optimize memory for Python and C

There are many reasons for memory expansion: Memory leakage in Python and C code, memory fragmentation, and low memory usage efficiency.

The low efficiency of fixed memory does not significantly help. They think there is a memory leak, but no.

The problem becomes memory fragmentation. Memory fragments occur in memory blocks of different sizes and are constantly deleted and allocated. What is the situation that continuous memory blocks can no longer be allocated. CPython does not have a garbage collector, so all the memory cannot be allocated and the heap memory keeps growing.

The solution is to create a custom distributor. The metadata object of this file has been increased and transferred a lot, so the obvious result is to use mmap in C to create a custom distributor.
 

Link: http://www.linuxidc.com/Linux/2011-04/34748.htm