Next-generation Big Data technology: Twitter releases Manhattan real-time database details

The emergence of the Twitter Manhattan database shows that today's database technology has been unable to meet the unique needs of large internet companies, and a new round of the database revolution has started.

Twitter is one of the fastest-growing products in all large internet companies, and this week Twitter posted the technical details of the Manhattan (Manhattan) database, the core engine of real-time data services. Manhattan is a distributed, real-time database that serves multiple teams and applications within the Twitter company.

The data processing power and speed of the Manhattan database dwarfs today's popular open source databases that have been unable to meet the Twitter-scale production environment, as Peter Schuller, a Twitter software engineer, said:

We wasted a lot of time trying to get the production system to meet the performance expectations of a variety of products. Configuring storage resources for a new application involves too much labor and process. According to our experience, the use of off-the-shelf database technology for a large web business such as Twitter is no longer feasible to develop and operate a production storage environment.

Schuller a blog detailing how Twitter developed a highly reliable, highly available, easy-to-use new generation of databases such as Manhattan. Details of some of the data formats designed for the Manhattan database are also published. Currently, users use the Manhattan database in the form of key-value storage, and in the future Twitter plans to add a user interface, including a graphical interface. The Manhattan database has three storage engines for read-only Hadoop data, a large amount of read data, and a large number of write data. In addition, there are a number of services built into Manhattan, including the introduction of Hadoop data, ensuring data consistency and computing time series data.

The most important thing for developers and engineers is that Manhattan as a storage service can be used like any other cloud storage service. Engineers can configure storage resources for applications in seconds, without having to wait for hardware to install or reconfigure the system architecture. Twitter has focused on ensuring that the overall performance of the Manhattan multi-tenant is not affected by the amount of resources a tenant is consuming.

Twitter plans to publish detailed technical development details in a white paper on Manhattan's database technology, which is scheduled to be released on the day. As a heavyweight fan of Open-source software, Twitter is likely to open up the Manhattan database technology some time in the future. In fact, Twitter had open-source its MySQL code in 2012 and recently worked with Facebook to develop a webscalesql open source database project.

The emergence of the Twitter Manhattan database shows that today's database technology has failed to meet the unique needs of large internet companies. Twitter, Google, Facebook, and LinkedIn have abandoned the original NoSQL technologies (which are entering large companies) to develop next-generation database technologies, as they developed Cassandra, Voldemort and BigTable. Perhaps Manhattan will be the Cassandra of the future, and LinkedIn's espresso become the new MongoDB (Editor's note: MongoDB frustrated in the developing market by gaining access to the NSA) and a favorite for the next generation of start-up developers.