Dockone Technology Share (40): Build an online development environment with Docker and Git

This is a creation in Article, where the information may have evolved or changed.

Disclaimer

The following share is an individual's experience in their spare time and does not represent IBM's point of view.

Why build a development environment in the cloud

Today, in the traditional development model, version control, automated testing, continuous integration has been on the ground to continue to the cloud.

• Test engineers can use automated test scripts to perform large-scale regression testing in the cloud
• Operations engineers can quickly deploy applications in the cloud with automated deployment tools
• Product owner, you can control the release of the product through the version control tool

But development engineers are also using the computer, notebook and other equipment to complete the development work.
In the project development phase, we usually face the following issues:

1. In the project start-up phase, how to quickly build and configure the development environment for development engineers to shorten project start-up time and cost.
2. In the project development phase, how to let the development engineer have:
   
   • The exact same development environment, to reduce the cost of debugging.
   • Architecture similar to the product environment to reduce the risk of deployment due to environmental differences.
   • Reduce equipment costs by reducing the performance and type requirements of development equipment.
   • Flexible allocation of development resources (CPU, memory, storage), so that engineers need a lot of development resources to quickly get resources.

Summary of issues to be solved in the cloud development environment

Based on these considerations, I've summarized the following 4 cloud development environment issues to solve:

• Ensure that every development engineer can quickly get a consistent "out-of-the-box" development environment
• Development environment to get rid of the limitations of development equipment, such as operating system, device performance
• Let the development environment simulate the structure of the product environment as much as possible.
• Flexible allocation of development resources

The challenge of building a development environment in the cloud

A "out-of-the-box" development environment should include: a fully functional code editor, an application runtime environment, and a debugging toolbox.

• To get rid of the limitations of the device, the integrated development environment can be implemented with mainstream browser-compatible technologies such as HTML5.
• In order to flexibly allocate resources and simulate the product environment, you can put the application operating environment in the cloud and use the container technology configuration.
• Debugging tools include the output of the log, breakpoints and other functions.

Summary of technical difficulties

• Online integrated development environment: compatible with major mainstream browsers
• Isolated operating environment: isolated, customizable and efficient
• Debug code: View logs to set trace breakpoints

Potential alternative technologies

Based on the technical difficulties, I have done some research and selected several alternative open source technologies.

• Online integrated development environment: Icecoder, Eclipse Orino, Codiad
• Isolated operating environments: cloud virtual machines, Docker
• Code debugging: Fluentd

Currently, only the use of Docker is determined to achieve the operating environment.

Integrating the online development environment, I use Git to decouple the code runtime and editing environment, and code synchronization.

Code debugging, I use a separate FLUENTD service to collect logs, log in to the virtual machine via SSH to see the Docker logs collected by fluent.

Architecture Overview for the Cloud development environment

The following diagram is a simple schema description. First, use Git to decouple the editing and operating environment of your code.
I'm in the cloud, using Docker to build a git bare library, and use git hooks to synchronize code in a bare library. Use Docker to run the app and check the results through the browser.

Because of the separation of code editing and running environments, development engineers have the flexibility to choose development tools. The development tool can be an online editor or a local editor.
With the flexibility of Docker, development engineers can quickly configure the application runtime environment and share it with development team members.

Concept Verification

• Edit the code locally
• Sync to the cloud with Git
• View results with Docker run code

A short video recording showing how to use the development environment in the cloud

Video shows how to develop a Django project in the Cloud

Detailed steps

Reference Online development Environment practice

Q&a

Q: So many developers, how to assign domain name?

A: At present, I am through the Nginx reverse proxy to assign different subdomains to different developers.

Q: How is development and testing linked in this system?

A: This system is mainly focused on the development of the part, the interaction between testing and development is not reflected in this system, but there are many formed DevOps system can be used as a reference.

Q: Is the cloud development environment in the cloud, the developer SSH into their own development environment, after the submission of the code compose is automatic or man-executed?

A: Ideally, the three parts will be put down to the cloud, at present I only realize the operating environment in the cloud. Compose is now a manual line, but can be automated by scripting

• A fully functional Code Editor
• An application run environment
• A debugging toolbox.

Q: Will cloud development reduce the developer's workload and make it more focused on the code rather than on the application of the process?

A: Cloud development will allow developers less experience to learn about middleware and spend more time focusing on the implementation of business functions, so that development can focus on the needs themselves rather than the technology itself.
In addition, the online development environment can improve collaboration. Because both the running environment and the code are in the cloud, developers can quickly switch other people's development environment and code.

Q: What is the experience of the online IDE in real-world use, and what is the level of acceptance by the developer and support for all languages?

A: The current attempt to online IDE is limited by the network, and the function is not completely compared with the local IDE. However, because of the decoupling of git, developers can choose to use the local IDE to edit the code

Q:git is still more complicated, is there a simpler tool?

A: In addition to Git, there is a SSHFS technology that can directly map remote code to Local. But considering the version of backtracking, and even future use of big data to analyze the developer's development behavior. Git can be a better choice. Git is now complicated by the fact that the corresponding automation scripts are not established, and developers can ignore the parts of git synchronization after they are built.

Q: After the code is submitted to the cloud, how does the container take effect, the code directory as a mount point or a process in the container in real time to pull?

A: Yes, the code is now attached to the container as a volume.

Q: But how can you ensure that the code is synchronized when the commit image is reproduced in a mounted way?

A: I think the question goes back to the question of "what is the system's concern". The system I am sharing focuses on the development part of the project and does not focus on versioning, testing, and deployment.
Typically, the project should have another Git repository for versioning, and then there will be a dedicated build system to automatically build the specified code.

There is no code in the mirror image of Q:commit? Do you intend to use any environment in a way that is mounted? Will this be a bad move?

A: First, the container is generated by the Dockerfile under the project path, but only once in the running environment. When development is over, it can be done through the project's dockerfile, overall build code and environment, and deploy.

Q: For example, with multi-branch parallel development such as git flow, does each branch have a corresponding Docker container? How do I associate branches and containers?

A: This is a good question, first of all, Docker only provides the code to run the environment, if the different branches of the code use a different operating environment, Docker container should be multi-set, and, different dockerfile should be with the code.
If the operating environment is the same, you can use the same set of Docker container. Just switch the code to container.

Q: Yourselves for the time being, it's really good to deploy on the cloud, but is the private environment for some production more reliable than the cloud?

A: I understand that the question should be whether to deploy the cloud development environment to a public cloud or a private cloud. My personal understanding is that, for the time being, it should be deployed on a private cloud inside the enterprise to avoid code leaks.

Q: At present, it is more convenient to build a unified environment using kubernetes or shipyard, and now the price of the physical equipment is cheaper and more controllable, is it really better than the internal control in the cloud?

A: As with the first question, the cloud can refer to a private cloud or a public cloud. I agree with the questioner's consideration that the private cloud will be better.

Q: Do you first commit the code, trigger the hook, synchronize the code in the container, and compile and execute. Would that be more efficient?

A: Decoupling the code editing environment from the running environment, using GIT to synchronize, the efficiency is constrained by two factors:

• I/O is the network speed and disk speed. This factor can be ignored if the network speed is guaranteed.
• The efficiency of git itself, because each time the save commits git,git itself is incremental storage, so the efficiency is not too low.

Q: Can I also take the same effect on a private warehouse? Can the hook mechanism also synchronize the code for a container that cannot be accessed externally?

A: The system is currently recommended to be deployed in the private cloud as a whole.

Q: Each developer has the same development environment, the development environment is using container, how to solve the network? That is, in the developer's eyes is the use of virtual machines, with a fixed IP.

A: It should be said that each developer has the same Code runtime environment, which is implemented through container. At present, there is no too much consideration for the network, in the planning there will be a network allocation components, for developers to assign different subdomains, and automatically route the domain name to container, so that more flexible allocation of host resources, management container.

Q: How does the code in the cloud guarantee security and prevent corporate code leaks?

A: Use a VPN when you are deploying on a private cloud and require access to a public network.

Q: My company's main need is C + + research and development of the telecommunications industry, not through the provinces, different models have to talk to a set of development test environment, you are talking about web development, C + + is suitable for the development of the cloud?

A: I mainly study the web direction. Just a simple idea, C + + Docker image is some, mainly to see what you need a debugging environment. If you can view the results over the web. I think it's doable.

Q: One question about using git to build a continuous integration environment is how to deal with the different situations of individual files in the development environment and production environment. For example, some configuration files, Django's settings.py, in the production and development environment, its content is different.

A: Typically, the configuration file is separate from the code, and with Django's example, you can use. env. In the development environment, and the production environment using different configuration files, rather than directly written in the code.
How to configure the product environment and development environment, you can refer to some good practice articles. Separation of the operating environment and code of practical articles, the industry is more agree with "The 12-factor App" This design idea.

Q: If the backend is managed by cluster mode, what is your estimation of the use of back-end container resources and the ability to load the entire cluster? Different projects, the impact on the backend resources is very large.

A: The project is still in the concept verification phase. For medium-sized Web projects, a single developer allocates 1CPU of resources, and 1GB of memory is perfectly enough.

Q: Local development can quickly switch the different code versions of local git to view, remotely run, want to see the previous code running results need to really put the code in the remote share this library rollback?

A: Yes, because each development has its own remote code codebase. Therefore, rolling back does not affect others, nor does it lose its own code.

The above content is organized according to the December 29, 2015 Night Group sharing content. Share people: Wang Yanxie, IBM Assistant Architect, is primarily responsible for Web application design and architecture. In recent years, focus on the impact of cloud technology on Web application architectures and development approaches. Start learning and focus on container technology from the beginning of 2015 and practice using GIT and Docker to build an online development environment in your personal amateur program. Github: @yanqiw, Twitter: @yanqiw, frankwang.cn. Dockone Weekly will organize the technology to share, welcome interested students add: LIYINGJIESX, into group participation, you want to listen to the topic can give us a message.