What is the difference between a Hadoop distributed file system and a OpenStack object store?

Recently, a question was raised on Quora about the differences between Hadoop Distributed file systems and OpenStack object storage.

The original question is as follows:

"HDFS (Hadoop Distributed File System) and OpenStack object Storage (OpenStack object Storage) all seem to have similar purposes: to achieve redundancy, fast, networked storage. What are the technical features that make the two systems so different? Is it significant that these two storage systems eventually converge? "

After the problem was raised, a OpenStack developer soon responded. This article extracts the first two replies for translation, for your reference.

The first answer comes from Rackspace's OpenStack Swift developer Chuck misspelling:

Although there are some similarities between HDFs and OpenStack object Storage (Swift), the overall design of the two systems is very different.

1. HDFs uses a central system to maintain file metadata (Namenode, name nodes), while in swift metadata is distributed and replicated across clusters. The use of a central metadata system is no different from a single point of failure for HDFs, so it is more difficult to scale to a very large environment.

2. Swift has taken into account the multi-tenant architecture in its design, and HDFs does not have the concept of a multi-tenant architecture.

3. HDFs is optimized for larger files (which is usually the case when data is processed), Swift is designed to store files of any size.

4. In HDFs, the file is written once and only one file is written at a time, while in swift the file can be written multiple times, and in the concurrent operating environment, the most recent operation.

5. HDFs is written in Java, and Swift is written in Python.

In addition, HDFs was designed to store a large number of medium sized files to support data processing, and Swift was designed to be a more generic storage solution that reliably stores a very large number of files of varying sizes.

The second answer comes from Joshua McKenty, the chief architect of NASA's Nebula Cloud Computing project, and one of the early developers of OpenStack Nova Software, currently a member of the OpenStack Project Supervision Committee, piston.cc, founder of the OpenStack based company.

Chuck gave a detailed description of the technical differences, but did not discuss the two imaginable fusion, OpenStack Design summit threw the topic of integration. In short, HDFs is designed to use Hadoop to implement MapReduce processing across objects within a storage environment. For many OpenStack companies, including my own, supporting Swift's processing is a goal on the road map, but not everyone thinks MapReduce is the solution.

We've discussed writing wrappers for HDFs, which will support the OpenStack internal Storage Application Programming interface (API) and allow users to perform Hadoop queries against that data. Another option is to use HDFs in Swift. But none of these methods seems ideal.

The OpenStack community is also doing some work on research and development, carefully studying other alternative mapreduce frameworks (Riak and COUCHDB, etc.).

Finally, there are other storage items that are currently "affiliated" to the OpenStack community (Sheepdog and HC2). Taking full advantage of data locality and making object storage "smarter" is an area that is expected to make progress.