Thomson Reuters-using multi-model database Arangodb to create fast and secure, concise view analysis

SUMMARY: Thomson Reuters provides professionals with the intelligence, technical and human resources they need to find the answers that are trustworthy. They enable professionals in financial risk, legal, tax accounting, and media markets to make the most important decisions, all supported by the world's most trusted news organization. This case was written by Tanvir Mansuri, chief developer of Thomson Reuters.

Technology is constantly changing the enterprise. We at Thomson Reuters are committed to leveraging technology to make the information we collect in our business processes more relevant and personal, while delivering information to our customers and employees more quickly. By using a shared platform and working in our business units, we want to make our data easier to access and penetrate into our people, regardless of channel.

To promote this approach, we want to create a complex Business analytics and intelligence (BA/BI) platform that gives all Thomson Reuters employees all the views. This is a considerable challenge because we must integrate many different data sources that contain semi-related data into different structures to meet the diverse needs of multiple departments and roles.

Choosing the right data store

For data access and management, it is clear that we need a fast, modeless data store to handle more and more unstructured data in our BA/BI applications. Our application uses more than 20 data sources to provide a variety of information. This requires a powerful query language that can express the broad range of questions our employees want to answer quickly.

A key requirement is to support ad hoc connections , as well as graph traversal , to use the correct data access strategy for different parts of the application, and to ask more questions. Our choice is open source code solutions, and a responsive community .

Why did we choose ARANGODB?

First, Arangodb is a true open source project with a developer-friendly Apache 2 license. In addition, the team behind it is very friendly and transparent. After a fairly short adjustment phase, we began to indulge in the Arangodb Query Language (AQL). For us, writing queries with AQL is straightforward, and we can take advantage of a variety of functions and data access patterns. The more admirable aspect of AQL is that it uses nested for loops to combine queries. Therefore, the conversion between writing code and writing queries using AQL is very smooth. Their "multi-model" scenarios and the possibility of local connection and chart traversal in AQL are also very good, as it is sometimes convenient to combine joins and traversal in the same query.

Another important is Arangodb's microservices framework Foxx. We use Foxx quite frequently; We have created more than 20 Foxx services for our applications. Frankly, Foxx is a bit difficult to get started with: Documents can be further improved, and more examples or best practices will be of great help. However, the current ARANGODB team and community support have eased this shortcoming. They are very professional and very responsive. This is one of the reasons we decided to use the ARANGODB database.

Currently, we store more than 270GB of data in Arangodb (dumps to 408GB disks). As the volume of data grows steadily, we will soon be moving to a three-node cluster, primarily to improve usability (see).

Our current single-node settings are 24 Vcpus and 512GB of RAM. For cluster settings, we plan to use the same machine for the primary device.
Our application itself is read/write intensive, with at least three writes per second and 2000 updates per second at peak times. Queues in our architecture help with Shadow writes, and can even load ARANGODB. The readings are generally stable and we expect no large peaks to be seen.

What did we get from the Arangodb?

First, we learned that combining different data models in queries is actually possible and sometimes very useful. queries run very fast and AQL is very intuitive, even if our product owners and business analysts are now writing huge queries that are fairly easy for more than 200 rows. Sometimes they have to ask which index should be used, but the performance of most queries is acceptable and does not require additional work.

The Foxx framework helped us greatly reduce our development time. As we will integrate with many rest services, we used to write a large number of emulators for integration testing. The rest service can now be derived from Foxx. We can also define our own routes so that the data that runs does not have to be sent to the customer, instead it can be processed in the database and then sent only to the customer. In this way, we can reduce a lot of trouble and improve security when needed. For us, Foxx and Arangodb provide great help, and the experience of using it is great.

In general, we can now get all the data needed--assurance, reporting, mobile, API portals--available in one place, providing fast and secure access. the AQL flexibility, as well?? The combination of different data models makes it easy to get the query run and optimization you need. thanks to Foxx and AQL, we have easily extended the functionality of our applications. This allows us to spend more time on actual application development and get the answers we need more quickly. After all, we are called the "answer Company".

The importance of key features

The original use case of acknowledgements Tanvir Mansuri.

Thomson Reuters-using multi-model database Arangodb to create fast and secure, concise view analysis