Distributed (i) Fix service registration and discovery

Background

Recently in the distribution-related work, because of insufficient manpower I can only come to godless alone; it's bad enough to look at the overtime schedule for this time.

However, there are a lot of interesting things to share in the future, the focus today is to discuss the registration and Discovery of services.

Problems with distribution

My business is relatively simple, just need to know what service instances are available now (not make remote calls, just get the information).

The simplest way to do this is to configure all of the service nodes in your app so that you only need to select one of the algorithms from the configuration list each time you use it.

But this has a very serious problem:

Because the application needs to flexibly adjust the number of service nodes based on the application load, my configuration cannot be written to death.

Otherwise, the new node will either not be accessed or the node that is already down has been requested, which is certainly not possible.

Often solving such distributed problems requires a common area to store this information, such as whether you can use Redis?

Each node starts with Redis registration information, and data is deleted when it is closed.

In fact, it is the storage node ip + port , and then need to know the service node information only to get to Redis.

As shown in the following:

However, this results in a frequent query of Redis every time it is used, so that we can avoid this problem by caching a copy of the latest data locally after each query. This gives priority to local access and can actually improve efficiency.

But there are new problems as well, and if the service provider's node is added or deleted, the consumer side is not aware of the situation at all.

The first thing to think about in order to solve this problem is to use scheduled tasks to update the list of services regularly.

The above plan must be imperfect and not elegant. The main points are as follows:

• Based on timed tasks, many invalid updates are caused.
• Timed tasks are cyclical and cannot be done in real time, so there may be request exceptions.
• If the service is forcibly killed, it is not possible to clear Redis in time so that this seemingly usable service will never be available!

So we need a more reliable solution, which is really similar to Dubbo.

The used classmates must be familiar with this picture.

Quoted from Dubbo official website

One of the very core content (red box) is the registration and discovery of services.

In general, consumers need to know the network address of the service provider (IP + port) in order to initiate a remote call, which is very similar to what I need above.

Dubbo, however, uses Zookeeper to solve the problem.

What can Zookeeper do?

Take a look at some of the key features of Zookeeper before discussing how to implement it.

Zookeeper implements a tree-like structure similar to the file system:

These nodes are called znode (what is called unimportant), where each node can hold certain data.

The main thing is that there are four types of Znode:

• Permanent node (node always exists unless manually deleted)
• Permanently ordered nodes (with a sequence number for each node at the end of the creation order, as follows: root-1 )
• Instantaneous node (the point exists when the client and Zookeeper are created, and is deleted when disconnected and notified accordingly)
• Instantaneous ordered nodes (added order based on instantaneous nodes)

What is the biggest question to consider using Redis above?

In fact, it is impossible to update the service provider's information in real time.

So how did you make use of Zookeeper?

Three main features: instantaneous node

Zookeeper is a typical observer pattern.

• Due to the characteristics of instantaneous nodes, our consumers can subscribe to the parent node of the instantaneous node.
• All instantaneous nodes are updated automatically when the node is added and deleted.
• The update notifies the subscriber of the latest node information when it initiates a notification.

This way we can get the information of the service node in real time, and we only need to cache it locally when we get the list for the first time, and do not have to interact with Zookeeper frequently, just wait for the notification to update.

This information is also deleted in Zookeeper regardless of the reason the node is down.

Effect Demo

This is how this is done.

To do this I created a new app for the demo:

Github.com/crossoverjie/netty-action/tree/master/netty-action-zk

is a simple springboot application, just doing a few things.

• App startup A thread is used to register the service with Zookeeper.
• Also listens for a node to update the list of local services.
• Provides an interface for returning an available service node.

I have started two applications locally: 127.0.0.1:8083,127.0.0.1:8084 . Take a look.

Two apps start complete:

visual tree structure for the current Zookeeper:

When you want to know all the service node information:

When you want to get an available service node:

Here just take a simple poll.

When a node is down: The app receives notifications to update the local cache. The nodes in the Zookeeper are automatically deleted.

When you get the latest node again:

It is also natural to get the latest information when a node is restored. The local cache is also updated in a timely manner.

Encoding implementation

Implementation is also relatively simple, mainly is the use of Zkclient API.

Stick to a few more cores.

Registered

Start the registration Zookeeper.

The main logic is in this thread.

• The parent node is created first. As shown in the Zookeeper node, the root node needs to be created before /route it is created to determine if it already exists.
• Then you need to decide if you need to register yourself in Zookeeper, because some nodes are just for service discovery, and he doesn't need to take on business functions (it's my own project needs).
• Register the current app's IP and port, and listen to the root node to be /route notified when the other services are offline.

Based on local cache

Monitoring service changes

    public void subscribeEvent(String path) {        zkClient.subscribeChildChanges(path, new IZkChildListener() {            @Override            public void handleChildChange(String parentPath, List<String> currentChilds) throws Exception {                logger.info("清除/更新本地缓存 parentPath=【{}】,currentChilds=【{}】", parentPath,currentChilds.toString());                //更新所有缓存/先删除 再新增                serverCache.updateCache(currentChilds) ;            }        });    }

As you can see here is the updated local cache, which uses the cache provided by guava, which is interesting to view before the source code analysis.

    /**     * 更新所有缓存/先删除 再新增     *     * @param currentChilds     */    public void updateCache(List<String> currentChilds) {        cache.invalidateAll();        for (String currentChild : currentChilds) {            String key = currentChild.split("-")[1];            addCache(key);        }    }

Client load

A load algorithm is provided at the same time on the client.

is actually a polling implementation:

    /**     * 选取服务器     *     * @return     */    public String selectServer() {        List<String> all = getAll();        if (all.size() == 0) {            throw new RuntimeException("路由列表为空");        }        Long position = index.incrementAndGet() % all.size();        if (position < 0) {            position = 0L;        }        return all.get(position.intValue());    }

Of course, this can be extended to more such as weights, random, LRU and other algorithms.

Zookeeper other advantages and problems

Zookeeper Nature is a great distributed coordination tool, and its features can be used for other functions.

• Data change Send notification This feature enables the unified Configuration center, and no longer needs to maintain the configuration separately in each service.
• Distributed locks can also be implemented by instantaneous sequential nodes.

In the realization of registration, the discovery of this demand, Zookeeper is not the most preferred.

Since Zookeeper selects CP (consistency, partition fault tolerance) in the CAP theory, it is not possible to get any data when half of the nodes in the Zookeeper cluster are unavailable.

There is no problem with consistency, but it is more recommended in the case of registration and Discovery Eureka , and has been verified in Springcloud. Specifically, it is not discussed in this article.

But given my usage scenario, Zookeeper is capable.

Summarize

All the complete code in this article is hosted on GitHub.

Github.com/crossoverjie/netty-action.

A seemingly simple registration and discovery feature is implemented, but distributed applications are far more than that.

Due to the network after the isolation of a series of problems that we need to improve in other ways one by one, follow-up will continue to update the distributed related content, interested friends may wish to continue to pay attention.

Your point of praise and forwarding is the biggest support.