Continuous Integration System Based on Java 9 module system and Vert. x

Continuous Integration System Based on Java 9 module system and Vert. x
Key points of this Article

• Vert. x is compatible with Java 9 and can be used together to build applications.
• Many Java class libraries still do not support modularity.
• Be especially careful about the "Automatic module" (some class libraries have not yet become modules ).
• Java's built-in Nashorn JavaScript runtime environment is very useful for Vert. x applications.

This article describes how to use Eclipse Vert. x to design and develop a message-driven, responsive CI system. We will use the Java Platform Module System (JPMS) to construct an application composed of multiple modules, and the modules communicate with each other through the defined interface.

With JPMS, architects and developers can use modules to reconstruct large legacy systems or use them to create new applications. However, it is not easy to use the existing Java class libraries in the module system. Therefore, we will also discuss various problems that may occur during the use of JPMS and how to solve these problems.

Let's first define the minimum available product (MVP) of this CI system, and we will build it into a Docker native system. This system must provide the following features and expose them through the rest api:

1. Supports CRUD operations for warehouses. A repository represents a project with the connection address of the Git repository.
2. Supports "pipeline as code ". The pipeline defines the Build Process and uses JavaScript to define it. The JavaScript script file can be stored together with the code.
3. Provides APIs for starting or disabling pipelines. An instance of the pipeline represents a building process.

After the MVP is defined, we can start to build our system. First, you must create the Project Skeleton. You can use the multi-module Gradle Project template provided by IntelliJ idea to create the skeleton. To use JDK 9, you are advised to select the latest version of Gradle (in this article, the latest version is 4.4.1 ). We also need to add the Jigsaw plug-in and set the code compatibility to Java 9. The main project file "build. gradle" should look like the following:

Like most other systems, we will have a public library for entity classes, tool classes, shared constants, and query Resolvers. We define this public library as a Java 9 module.

As mentioned earlier, the Java 9 module is a collection of interfaces, classes, and resource files. It has the characteristics of self-description and has its own name. JPMS introduced the "module-info.java" file, which developers use to define the public contract of the module and the dependency on other modules. We will also use this file to name our module and specify the dependencies on other modules and the Public packages exposed by the module itself.

Is the sample code for the module-info.java file:

Each module-info.java file starts with the keyword module, followed by the name of the module. The Reverse Domain Name naming method used for package naming can also be used for module naming.

The code block contains two new keywords: "exports" and "requires ". "Exports" is used to declare the public package exposed by this module, that is, the public API of the module. "Requires" is used to declare dependencies on other modules.

So the problem arises. What if the dependency package of a Java 9 module is not a module? At this time, the automatic module will be used.

As its name tells us, non-module JAR packages are automatically converted into modules, and the module name is generated based on the JAR package name. The generation of the module name follows the following rule: It starts with a JAR package file, removes the extension, and replaces the hyphen with a dot. If a version number is available, the version number is removed. In this case, the module name corresponding to vert-core-3.5.0.jar is vertx. core ". However, this method may not always work. Next we will give an example related to the Netty dependency package.

In addition to core modules, we also need to define other modules for accessing databases, user authentication, running engines, and interacting with other plug-ins in the CI system.

After introducing some concepts of Java modularization, let's talk about Vert. x. Vert. x is a tool kit that provides non-blocking APIs. That is to say, Vert. x applications can process a large number of concurrent requests with only a few threads. Vert. x adopts the multi-reactor mode to achieve this goal.

Developers familiar with JavaScript may still remember the single-thread event loop model. The multi-reactor mode is similar to this mode, except that it uses multiple threads. Vert. x creates a number of event loop objects based on the number of CPU cores of the given server.

Vert. x also provides an actor-based concurrency model. In the Vert. x ecosystem, actor is called "verticle". verticle communicates with each other through JSON messages, which are transmitted through the event bus. We can also specify the number of verticle instances to be deployed.

The event bus can be a cluster managed by the cluster manager, such as Hazelcast or Zookeeper. We can regard verticle or verticle combination running on the Vert. x instance as a microservice. The mutli-reactor model, verticle, and event bus give Vert. x applications high responsiveness and elasticity. Therefore, we can say that Vert. x applications are reactive.

Now let's take a look at the overall process of this CI system:

As shown in, several verticle communicate through the Vert. x event bus. Note that the plug-in the figure is also verticle. Server verticle is the entry to the CI system and exposes a rest api. The command line or GUI client can use this API to specify the connection address of the code repository, create and run the pipeline.

The following code defines APIs and routes in Vert. x:

We use the Web class library of Vert. x to define the rest api, and all routes are prefixed with "/api/v1. Vert. x also provides many other class libraries for rapid development of reactive applications.

For example, we can use the Web API class library to design an application API Based on OpenAPI 3. This class library will help us handle request verification and security verification issues. Vert. x's OAuth class library can be used to improve the security of application APIs. OAuth vendors can be Google, Facebook, or custom.

In the previous image, Engine verticle coordinates the pipeline execution. After the client calls the API provided by Server verticle, Server verticle sends a message to Engine verticle. Engine verticle initializes a new flow object after receiving the message.

The flow object is actually a simple state machine used to track the execution status of the pipeline. At any time, the flow object may be in one of the three States: setup, run, or teardown. It changes the status based on the input message. When entering a new state, the flow object will trigger an event and send the event to the event bus.

The plug-in registered on the event bus will process these messages and asynchronously return the processing results through the event bus. The following code demonstrates how to register a message processor, create a flow object, and process incoming messages:

Engine verticle is also responsible for locating and deploying other plug-ins or verticle. We use the service loader mechanism introduced in Java 6 and improved in Java 9 to locate and deploy the plug-in during server startup. To better understand the service load mechanism, it is necessary to discuss the service and service provider.

A service is actually a known interface or class (usually an abstract class), while a service provider is a specific implementation of the service. The ServiceLoader class is used to load the service provider that implements a given service. We can declare in the module that it uses a specific service, and then use ServiceLoader to locate and load the service provider deployed in the runtime environment.

For example, the server module declares that it needs to use the Plugin interface, and the workspace module declares that it will provide two services that implement the Plugin interface.

Therefore, when the server module is started, it will call ServiceLoader, find two ins, and deploy them into verticle:

The plug-in completes a lot of work, including registering a message processor to process MPs queue events of interest. For example, the workspace plug-in is responsible for synchronizing Git code, while the script parser plug-in is responsible for scanning the workspace and identifying and executing pipeline script files (written in JavaScript ). Some shell commands are generated after the script file is executed, and the script executor plug-in the Docker container will execute these commands. Because Vert. x uses the Nashorn engine built in Java, JavaScript code can be fully run. You know, Vert. x also supports JavaScript, Kotlin, and Groovy.

The following is some code of the pipeline script file:

After receiving the message, the script executor plug-in downloads the Docker image, creates a container, and runs shell commands.

Docker REST APIs are usually exposed through the unix domain socket-based HTTP, rather than the traditional TCP socket-based HTTP (S ). This is also one of the places where Vert. x can play its role. We can use the asynchronous client of Vert. x to interact with Docker, rather than using a common synchronous blocking scheme.

If the project contains the native transmission class library provided by Netty, Vert. x will use native transmission instead. This happens if we specify a dependency like netty-transport-native-kqueue In the build. gradle and module-info.java files.

The next version of Vert. x may support HTTP Based on unix domain socket. Currently, we can solve this problem by modifying a small amount of code in the Vert. x class library. The plug-in code that interacts with the Docker engine looks like this:

Adding a non-module JAR package "netty-transport-native-kqueue-4.1.15.Final-osx-x86_64.jar" as a dependency will automatically generate a module name, but because JAR contains the Java keyword native, our application cannot be compiled normally.

Netty will solve this problem in the next version. Before the problem is solved, we can add "Automatic-Module-Name" to the manifest file of the JAR package to bypass this problem. Therefore, we need to decompress the JAR package and modify "MANIFEST. MF file, add "Automatic-Module-Name: io. netty. transport. kqueue, and then re-package through the following command:

The following command can be used to verify whether the specified automatic Module name in the manifest file can be correctly identified:

We also need to use the same command to solve the naming problem of other non-module JAR packages.

Then we can build and run our CI system. The following is a command to run the application:

To support JPMS, we added some new parameters in the "javac" and "java" commands, which tell the Java compiler and runtime, use the module path and the module JAR package to replace the original class path.

Some noteworthy parameters:

"-P" or "-module-path" is used to tell the Java system to find the Java module in the specified directory.

"-M" or "-module" is used to specify the module and main class.

In this article, we have designed a modular microservice Application Based on Vert. x. We used JPMS and JDK 9 and built a Docker native CI system. You can download the relevant code on GitHub to learn how to develop a small self-contained modular Java application based on Vert. x and the modular system.

About the author

Uday Tatiraju is the chief engineer of Oracle and has 10 years of development experience in e-commerce platforms, search engines, backend systems, and Web and mobile programming.

 

 

Building a CI System with Java 9 Modules and Vert. x Microservices

This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151252.htm