Mercedes-Benz's DEVOPS practice in building future cars

Siva Mandadi is a devops engineer from the user interaction department at the Mercedes-Benz North American Research and Development Center, where he shared the Jfrog User conference to introduce Mercedes-Benz's experience of building a future car through DevOps. This article is based on video data, the original video address:

Https://www.youtube.com/watch?v=zw0QsPgD5Ug

Background introduction

Mercedes-Benz's definition of the future car covers many aspects, including in-car music, navigation, autonomous driving and so on, related to the implementation of technology is also multifaceted, such as machine learning, the Internet of things and other cutting-edge technology. These technologies have a "sense of the Future" in practical applications. For example, in the future the car can change the theme of the car according to the weather conditions, such as color, music, etc., so that the owner is always comfortable in the state of the moment, or, if the owner is driving home, half an hour away from home, can send messages to home kitchen appliances, began to prepare a delicious meal.

At the beginning, the entire user interaction department (the video presenter's department) had about 300 engineers, from different countries, in different regions such as the United States, Germany and India, and the team performed approximately 60 build tasks per day, with more than 1000 self-research pieces of about 1TB of data per week; average CI The build duration is 6 hours.

The challenges facing

With the development of the business, the team is faced with great challenges, mainly in the storage and sharing of artifacts, dependency management, orchestration, continuous construction and warehouse different aspects:

Self-research storage: More than 1000 self-research pieces are produced per week, requiring nearly ITB storage space, and the existing storage resources are quickly depleted over time.

Self-study sharing: Team members around the world, upload and download speed becomes a bottleneck, development and QA staff will often complain that the download speed is too slow. For example, in India and Germany developers need to upload, the U.S. side of the QA need to download the test will encounter this situation.

Dependency Management: The dependency management of a large research and development team is also a very headache, some feature-dependent packages have different versions in different modules, and maintaining complex dependencies is a very challenging thing. The team primarily uses Java and C + + development, and uses Maven, Conan as a dependency management tool.

Continuous integration: Continuous integration is necessary for such a large team, and each submission must be ensured through continuous integration (CI) to ensure that the submission does not pose any problems. The team used Jenkins and Bamboo to do some continuous integration related things, and it's worth mentioning that Docker Container was used as the Jenkins agent to dynamically create the build Agent based on the load through the configuration file. However, because the Docker mirrors are very large and in different geographies, the process is less efficient.

In the face of the above challenges, in the CI construction phase, the download depends on the time to reach 5-6 hours, which is unacceptable, and because the Artifact is very large, it will make the whole system run slowly. One solution is to use NAS storage. But there are still some new challenges:

The price is expensive.

NAS storage can solve problems to some extent, but the related equipment and maintenance costs are very expensive.

Number of builds: because the built-in self-study grows very fast, storage space is relatively limited, so you need to limit 10 builds in real-world situations.

Sharing: Problems with sharing are still difficult to solve, especially across geographies.

Solution Solutions

Following the analysis of the challenges, it is considered that there are two alternative solutions. One is to add storage, and the other is to use a Nexus, but with the support of a highly available architecture and REST API.

Both of these options undoubtedly require a powerful DevOps tool. For multinational research and development teams, high availability is a very important feature, multi-language warehouses are also required, and increasing storage is obviously difficult to solve the underlying problem. The Nexus has multiple language support, but requires additional maintenance for both high-availability and REST APIs. After researching a number of related products, Mercedes-Benz Research and development team think Jfrog Artifactory is a very good choice.

Jfrog Artifactory offers a wide range of features, such as a unified warehouse, consistent UI, enterprise-class high-availability support, YAML-based predefined configuration, rich REST API support, and good customer support, among others. Artifactory can seamlessly connect to open source tools, especially Jenkins, so the user's old habits hardly need to change, and after use, the entire DEVOPS environment has changed dramatically.

In addition, Artifactory has many very powerful features, such as the DevOps full lifecycle metadata record, the AQL (artifactory query Language) query Language, cross-language service forward and reverse dependency resolution, and Xray Vulnerability scanning and impact analysis, and so on.

Mercedes-Benz through the artifactory to achieve a unified dependency management and centralized package warehouse, combined with its strong positive and negative dependence analysis, easy to solve the problem of unclear dependency relationship. Artifactory's geo-replication solves the performance bottleneck of package uploads and downloads, and because of the caching mechanism, the build time is significantly reduced, although the build frequency increases 10 times times per day. In the case of self-developed storage, storage optimization with Artifactory saves a lot of storage space, and through automatic scavenging mechanism, many management costs are reduced, and the risk of human operation is reduced.

The change of roles of the globally distributed Research and Development center and personnel makes the control of the authority become very complicated, because artifactory centralized management of the binary package of all languages, Mercedes-Benz in the use of Artifactroy easily after the implementation of the local developers of fine-grained permission control, Sharing within a team can also control permissions in a way that ensures that the minimum permissions model is always met.

From the data we can see that the number of daily builds has increased by 10 times times, the artifacts generated 10 times times a week, but the storage has only increased twice times, storage optimization is very significant. The most obvious effect is that the time to build CI is reduced by 3,600%, which is amazing. Of course, these are intuitive effects, and some are potential benefits, such as new people familiar with the entire project time is shorter, the development cycle is shorter, more accurate access control, the corresponding security also greatly improved.

Summarize

Mercedes-Benz's challenges exist in many companies, especially large multinational development teams. Enterprise-class high-availability, multi-language centralized warehouse, cross-region replication distribution and strong dependency management functions are necessary to meet the needs, Jfrog Artifactory in the Mercedes-Benz practice to provide a practical solution to achieve the smooth landing of the DevOps, with reference significance.

Author: Shanhui, Jfrog Senior engineer, focused on software system architecture and Enterprise IT transformation, has been in HPE as a cloud computing technology consultant, container architect, DevOps architect, etc., has served the customers in the oil and Sinopec, Motorola, Alcatel-Lucent, Huawei and so on. Good at container and container orchestration technology, familiar with hybrid cloud architecture, micro-service architecture, continuous integration and delivery, DEVOPS, etc. Long-term participation in open source community development and translation work.