Gleasy chief architect Shike: Implementing distributed framework and middleware based on open source

Note: This article starts in CSDN, reprint please indicate the source.

"Editor's note" in this article for the online Office platform Gleasy co-founder, technical team boss Shike wrote, he gave us to share the gleasy all the way to the technical combat. It is reported that the release of nearly three years of Gleasy, has successfully accumulated more than 50,000 enterprise users, in response to mass storage and high concurrency under the premise of a variety of basic problems in the solution of deep experience.

At the same time, March 18, 2015, Gleasy will officially release the 3.0 edition of "about", and the introduction of the English version, the traditional version, and the launch of the Internet Service Partner Solicitation program.

The following is the text:

Gleasy as a cloud technology service provider, the main solution to two key issues, one is the application of integration issues, that is, the need for specialized development of the industry application integration into the Gleasy platform, so that these from the user level is completely gleasy part of the application can also be seamless integration of communication. The second is the technical framework problem, that is, how to simplify the application development complexity through standard framework and middleware, and avoid application developers to reduce the application development life cycle by using a large amount of input on some key and in-depth technical issues (such as distributed, concurrency, mass storage, high performance, high availability, etc.).

Application Integration

Gleasy uses a process communication technology called online application to resolve application integration issues. Gleasy inside running a plate, said, the United States Mito and so on, are all a separate application. When users use Gleasy, how these standalone applications open, how resources are loaded, how they are managed (such as close), how to monitor, how to invoke (such as opening a file with the United States Mito) and so on, are the problems solved by this technology.

Application process: Applied in the Gleasy runtime, it can be a URL (such as the United States Mito), can also be JS class and Dom (such as a disk). When a user installs an application, the registration entry is generated in the online application registry, which contains the applied URL or the JS class (main function) for startup. When you click on the icon to start, the kernel Process Manager (Processmanager) reads the registration entries, dynamically loads the resources (JS, etc.) that are required to run the application runtime, and runs the main function to generate an application process instance.

Interprocess communication: The application declares the services it can provide by invoking the APIs provided by the process Manager. The caller invokes the service by invoking the Process Manager's API. For example, select a file from the project management of this call, the process Manager will look for the "one disk" application process, see if it exists, and if it does not exist, try to start it, post the event to the process's message queue when it is successful, and a single application process will periodically consume messages in its own message queue. In order to realize communication between applications.

Based on process communication technology, Gleasy declares a large number of services that can be invoked for all applications. By invoking these services, applications can easily integrate features such as single sign-on, select an organization structure, open/save files, initiate mail, initiate workflows, initiate schedules, and initiate chats.

Technical Framework

One of the challenges of cloud technology is how to deal with the fundamental problems of mass storage and high concurrency. Some seemingly very simple and basic questions, once the massive storage and high concurrency, it becomes complex, such as long link maintenance and message push, file storage, real-time retrieval, timing tasks, and so on, if not in the framework of the full planning, and so on when the system molding to adjust, will be a disaster. Gleasy after a long groping, in the open source and independent research and development program between the appropriate balance between the end of the open source, the absorption of several long, using a variety of technologies to achieve a full set of distributed framework and middleware to meet these challenges.

Distributed Message Queuing (CLOUDMQ): The key to solving the bottleneck of system traffic, using MQ can easily transform some hot spots into asynchronous implementations, thus fundamentally improving performance. Reference to the Apache Open source project Kafka and Taobao Open source project Meta. Inherits the distributed and High-performance high throughput features, but discards the broker design of the first two, making the whole scenario lighter. Based on Java to build, with the help of Redis to realize the queue storage, with the help of zookeeper to achieve node coordination and production consumption scheduling.

Distributed File System (CLOUDFS): Absorbing the experience of HDFs and Fastdfs, especially for reference to the idea of the group of Fastdfs, but introduced such as automatic to heavy, file speed replication, breakpoint transmission, etc. more suitable for the application of cloud OA class features. Based on Java NIO implementation, with the help of CLOUDMQ to realize the real-time synchronization of files between nodes, with the help of Redis Storage file node index information, realize automatic weight.

Distributed Instant Messaging (Cloudim): OpenFire is very good at long connection maintenance and message push, the universality of XMPP protocol makes it very convenient for multi terminal development. But OpenFire does not support distributed, and the MySQL based storage technology is daunting in terms of performance. Fortunately, it provides a perfect plugin mechanism, so that we create a fully suitable for their own distributed IM convenience. Based on the plugin mechanism, the storage is completely replaced with Redis, the zookeeper is used to coordinate the nodes, and the HTTP interface is provided for the backend system integration. A distributed Instant Messaging middleware is built in this way.

Distributed retrieval Platform (CLOUDINDEX): Data retrieval, especially large data retrieval, inadvertently will cause the entire system into a crash, the CPU easily up to 100%, even if the index, insert and update is also a nightmare. Based on SOLR, Lucene provides a rich and convenient data quasi real time retrieval scheme. But the master-slave replication and snapshot mechanism of SOLR, after our long verification, found that there were obvious problems with large data, one being latency, and the other was multiple crashes (I/O Hu Shen). But under the transformation, the introduction of CLOUDMQ using Message queue for data synchronization between nodes, read and write performance has been greatly improved.

Distributed task Scheduling (Cloudjob): Timed tasks are essential to many functions, such as calendar reminders, birthday reminders. Quartz in a stand-alone environment performance is still possible, but once involved in N machine timing task trigger, it is more difficult, performance does not go, often will not receive reminders. Redis rich data structure HashSet for our task queue to provide a great convenience, combined with the zookeeper node coordination capabilities, a sophisticated and powerful distributed task scheduling is produced.

Gleasy has been on the road for the entire technology service provider for 4 years, and has accumulated a lot of experience and lessons. In the process of distributed design, due to the different function characteristics and requirements, alternative schemes can vary widely. To understand a technology, the most important thing is to understand the limitations of the weaknesses behind the technology. As listed above, some of the core middleware of independent research and development, initially started from open source, and finally ended up with an independent research or two-time transformation. But if there are not many open source products at the beginning, it is estimated that the detours we take will be longer and more rugged. Technology selection, you should believe that there is no best technology, only the most appropriate technology.

The author introduces: Shike, Gleasy co-founder, head of technical team, senior Internet architect, is currently responsible for gleasy all product development and operation, research and development team construction and management, and as the chief architect, responsible for the Gleasy platform of the underlying architecture related design work.

by the "2015 OpenStack Technical Conference", "2015 Spark Technology Summit", "2015 Container Technical Summit" of the composition of the OpenCloud 2015 Congress in April 17-18th, held in Beijing. The schedule is all open! The knowledgeable people are here! (
)

"Manuscripts proposal" in recent years, cloud computing has become the core driving force for enterprise transformation, not only the enterprise's IT Infrastructure management brand, but also to bring more intelligent operation. At the same time, cloud computing is leading the industry to change the way people live. Although the cloud computing in the rapid development, but from the start, research and development, application, promotion and other aspects, in China, the development time is relatively late, which has many problems, can be described as opportunities and challenges coexist. In order to promote cloud computing technology development, better service providers, CSDN cloud computing channel for the cloud service provider, start-ups and individuals, such as collection of manuscripts, invites the company's doers to talk about your technical combat, as well as case-sharing, but also welcome a hot topic or technology to share your views.

Submission + consultation please email: qianshg@csdn.net.