I want to do some Python project automatic launch deployment tools, but the Python package dependency cannot be directly decompressed and changed to a configuration file when packaging and deploying specific jar files like Java, you must install the dependent modules one by one before deployment. This means that the efficiency is low and there is a high chance of errors during the installation process, I want to know how all Internet companies are working and what mature solutions are available? I want to do some Python project automatic launch deployment tools, but the Python package dependency cannot be directly decompressed and changed to a configuration file when packaging and deploying specific jar files like Java, you must install the dependent modules one by one before deployment. This means that the efficiency is low and there is a high chance of errors during the installation process,
I want to know how all Internet companies are working and what mature solutions are available? Reply:> the Python package dependency cannot be decompressed directly when the specific jar is packaged and deployed Like Java.
To some extent, this can be done. Similar to the Java jar package, Python has a type of egg package for a long time (in fact, the wheel package is also acceptable, but the tool I will introduce later does not support), as long as it is placed in sys. the path can be directly imported.
For example, My GitHub-youngking/buildout-package: My Custom buildout script
Refer to the bin/buildout writing method. You can run bin/buildout directly in the project wherever the code is dragged.
The zc. buildout tool can automatically encapsulate the project above. This tool can download all the dependent packages of the project and compress them into egg packages. By default, the egg package is placed in the eggs directory of your project, and a bin directory is generated. for various commands generated by the entrypoint defined by py, buildout will modify the sys. path to add the dependent egg.
Assume that your project was previously like this:
project/ setup.py src/ tests/ .....
- Docker is not a silver bullet. Development is too lazy, and O & M and platforms need to pay off their debts. If you want to use it in a production environment, your team needs mature O & M personnel, engineers with the ability to track upstream bugs, and technical pitfalls that can be extracted from business needs. Therefore, the small team of a small company should use it with caution.
- CopyVirtualenvDeployment is feasible, but the portability is poor. FirstVirtualenvAbsolute path change is not allowed.VirtualenvSome installed packages may depend on the dynamic library of the system (such as OpenSSL), and direct replication may encounter ABI compatibility problems.
- For Python project deploymentSetup. pyPackaging is a good choice. Generally, onlyRequirements.txtNoSetup. py, You can writeSetup. py, WhereInstall_requirePartially readRequirements.txtData
Use
Setup. pyTo package a Web application, you need a CI server, including Travis CI, GitLab CI, and Jenkins.
Setup. pyRolling release is not supported by default, so read
JOB_ID,
BUILD_IDOr use the current date and time to generate a CI environment variable. In the CI service, you can set the master warehouse
MasterAfter passing the test, the system automatically sends a version to the internal PyPI (If Jenkins is used, you can also use the static file service provided by Jenkins ). Select a version in the internal PyPI during deployment,
Pip installTo the production environment node.
Based on
Setup. py, You can use
Http://MANIFEST.inTo the release package, for example, the front-end static file generated by gulp compilation. In addition, if other projects (such as independent administrator backgrounds) depend on the current application and the service is too troublesome, you can directly reference the current application as the dependency to the internal PyPI. Such as avoiding manual Modification
Sys. pathHack that is highly bound and cannot be transplanted.
I have tried this method in my current company, and it seems that I have not encountered any problems. And
WheelAfter the package, the deployment speed becomes faster (saving the time required to compile additional resources during deployment ). The configuration we are using is posted here for reference: setup. py-GitHub
Other information:
- If you need to set up Intranet PyPI, devpi is recommended.
- Setup. pyOfClassifiersRemember to joinPrivate: Do Not UploadPublic PyPI will reject the inclusion of packages with this tag. This prevents mengmeng teammates from passing the company code to the public network PyPI.
- If you compress other project DependenciesWheelPublishing to intranet PyPI saves a lot of time for building and compiling. But be careful,WheelThe format is currently not ABI compatible (this is also public PyPI. Currently, only versions of Windows and OS X are allowed.Wheel), Please try to make sure that the Linux Release Used in the packaging environment (that is, the CI server) is consistent with the production environment.
- When configuring CI, you should not submit the Intranet PyPI logon password to the version library. Most CI Systems Support private environment variables
- No needSetup. pyFor packaging, the production environment of the Python project should be deployedVirtualenv, Even if one machine only deploys one application.VirtualenvAt least project dependencies and Linux releases can be isolated.
Consider turning the dependency into a wheel package for offline installation? I browsed the answers from all of the above and talked about the Python deployment and release process of our team based on my actual experience in the project.
Many companies are still using
Stone AgeHow to do this?
1. Write code locally. There may not be any virtualenv environment yet. Yes, this is actually the case in my old house. 2. write a script to install the global environment of the system. For example, MySQLdb may also use apt-get or yum to install python-dev and other system dependencies, and then use pip to install Python dependencies. 3. submit it to svn/git, pull the code on the test machine, and run the script to install the dependency. If it is a web project, it may be directly python web. run the py 8080 command to test whether an error will be reported. After testing several interfaces, you can find that there is no problem. Turn off the test machine. 4. pull down the code in the production environment or deploy the system. The deployment system here is generally a web page that can package the code on svn/git and then execute a script, to complete the corresponding deployment, it may also be executed directly on the machine: nohup python/path/to/python/main. py 2 & 1>/dev/null & start this process, and you may also have some business monitoring to regularly monitor the survival status of this script. 5. nginx may have been configured. If you publish a Django application, open the browser and check that the webpage is correct.
For Internet applications, the best solution to dependency problems is static Composite packaging-I am referring to ideas rather than specific technologies.
According to this idea, you can either solve it at the language level. For example, golang, if it is another language, use docker to answer @ squirrel Oreo. It seems that we are talking about docker. Here I also talk about docker. After all, our company is also heavily dependent on docker.
First, let's answer the questions @ alexsunmiu. Obviously, this old man's docker posture is incorrect:
1. Docker has two main concepts: container and image, with different responsibilities. The image is responsible for packaging the runtime environment, while the container is the runtime packaging. As for the mainstream old operating systems mentioned above, this is a specific business dependency and Docker cannot be used. However, please do not assert that the old systems are mainstream, each factory is different.
2. Container porting and packaging are actually very convenient. The large capacity you see is just something on the surface. The underlying design of Docker is highly reusable. A single image is stacked together. For example, you have a docker image, which contains ubuntu + python and ubuntu + java. If the two are of the same ubuntu version, in this case, layers are reused. If you have already used this ubuntu image, this part will not be downloaded again, but will only download the corresponding python and java parts.
3. Docker is isolated through cgroup. Although the security is not as high as that of the kernel-level isolated VMS, it is not so easy to gain control of permissions.
4. It takes so long for you to apt because you have not replaced the sources. list in the original ubuntu image, and you will naturally be banned by the great GFW. In fact, China's national conditions determine that no matter what external tools are used, we have to make changes first. Therefore, the docker factory will first create a private registry and official image, and then build several basic docker images based on the factory environment. All subsequent apps are based on these basic images.
5. dockerfile is not designed for shell use. It is used to define docker image building information. You have complicated building logic, which can be encapsulated into specialized scripts, and then ADD it to run it.
O & M focuses on Docker mainly in the traditional aspects of stability, monitoring, and logs. These are linked to the O & M KPIs. This is because it is still quite new, and features are gradually increasing. There are still a lot of hidden bugs, except that there are enough bugs, and some specific scenarios (the most common is the network part and the log Part. Not every problem can be resolved in a familiar way. So when you encounter these problems, it is very important for the group to be able to track bugs, or even to directly modify the code, because this ensures that the problem is controllable, otherwise, if something goes wrong one day, everyone may be confused. The year-end bonus will be ruined, and the roll-out will be taken away.
=== Update: 2016-01-15 ===
Add @ squirrel Oreo's comment:
In addition to O & M problems, the container also has a point like a giant "full static link", so security is also a concern ...... Previously, coreos developed clair to check the images on quay. io. A bunch of images have published CVE security issues ......
Security is also a problem. No one mentioned zc. buildout. AWS Codedeploy ...... In addition to copying the virtual environment package as a whole, the dependent package can also be packaged and deployed together with the site package ...... Pyenv + self-built pypi source, no sei.
