Don't talk about Hadoop. your data is not big enough.

Written by DontuseHadoopwhenyourdataisntthatbig ": ChrisStucchio, a data scientist with many years of experience, a postdoctoral fellow at the corang Research Institute of New York University, worked as a CTO of a startup company on a high-frequency trading platform, he is more often referred to as a statistician. By the way, he has started his own business and provided data analysis and recommendations.

Don't use Hadoop when your data isn't that big"

Author: Chris Stucchio is a data scientist with many years of experience. he is a postdoctoral fellow at the corang Research Institute of New York University and has worked on a high-frequency trading platform. He is also a CTO of a startup company and is more accustomed to calling himself a statistician. By the way, he now started his own business, providing data analysis, recommendation optimization consulting services, his mail is: stucchio@gmail.com.

"How much big data and Hadoop experience do you have ?" They asked me. I have been using Hadoop, but I seldom process tasks of several terabytes or more. I am basically a newbie to big data-I know the concept and have written code, but I have no large-scale experience.

Next, they will ask, "Can you use Hadoop for simple group by and sum operations ?" Of course I will, but I will say I need to look at the specific file format.

They gave me a USB flash drive with all the data in it, 600 MB, right, all of them. I don't know why. they are not satisfied when I use pandas. read_csv (Pandas is a Python data analysis library) instead of Hadoop to complete this task.

Hadoop is actually quite limited. It is nothing more than running a general computing, expressed as: select g (...) with SQL Pseudocode (...) FROMtable group by f (...) you can only change the G and F operations unless you need to optimize the performance in the intermediate steps (this is not very fun !). Everything else is dead.

(For more information about MapReduce, I have previously written an article titled "understanding MapReduce with 41 words .)

In Hadoop, all computations must be written according to a map, a group by, an aggregate, or such computing sequence. This is the same as wearing a tights. Many computing models are more suitable for use. The only reason to endure tights is that it can be extended to a huge dataset. But your dataset is probably far from that order of magnitude.

However, because Hadoop and big data are buzzwords, half of the people in the world want to wear tights even if they don't need them.

But I have hundreds of MB of data! Excel cannot be installed

Excel is very big, but not big data. There are many good tools-I like to use Pandas based on Numpy. It can load hundreds of MB of data into the memory in an efficient vectorized format. in my three years old notebook, Numpy can complete 0.1 billion floating point calculations in a blink of an eye. Matlab and R are also great tools.

Hundreds of MB of data is usually read, processed, and written to a file row by row using a simple Python script.

But I have 10 GB of data!

I just bought a laptop. 16 GB memory costs $141.98, and 256 gb ssd overwrites $200. In addition, if you load a 10 GB csv file in Pandas, actually, it is not that big in the memory-you can store a numeric string like "17284932583" as a 4-or 8-digit integer, and "284572452.2435723" as an 8-bit dual-precision.

In the worst case, you can load all the data to the memory at the same time.

But my data is 100 GB/500 GB/1 TB!

A 2 TB hard drive costs $94.99, and 4 TB is $169.99. Buy one, add it to the desktop or server, and then install PostgreSQL.

The applicability of Hadoop is far smaller than that of SQL and Python scripts.

In terms of computing expression capability, Hadoop is much worse than SQL. Computing that can be written in Hadoop can be easily written in SQL or simple Python scripts.

SQL is an intuitive query language without many abstractions. it is commonly used by business analysts and programmers. SQL queries are often very simple and usually very fast-as long as the database is correctly indexed, queries that take a few seconds are not very common.

Hadoop does not have any index concept. it only knows full table scan. In addition, there are too many abstract layers of Hadoop-my previous project was able to cope with Java memory errors, memory fragments, and cluster competition, and the actual data analysis work had no time.

If your data structure is not in the form of an SQL table (such as plain text, JSON, or binary), it is generally easier to write a small Python or Ruby script for row-based processing. Save it in multiple files and process them one by one. When SQL is not applicable, Hadoop is not so bad in programming, but it is still inferior to Python scripts.

In addition to being difficult to program, Hadoop is generally slower than other technical solutions. As long as the index is used well, SQL queries are very fast. For example, to calculate join, PostgreSQL only needs to view the index (if any) and then query each key required. For Hadoop, a full table scan is required and the entire table is rearranged. Sorting can be accelerated by sharding between multiple machines, but it also brings about overhead of cross-machine data stream processing. To process binary files, Hadoop must repeatedly access namenode. A simple Python script only needs to repeatedly access the file system.

But my data exceeds 5 TB!

Your life is really bitter-you have to go through Hadoop, and there are not many other options (you may also be able to use many high-tech machines with hard disk capacity ), in addition, other options are often too expensive (IOE and so on appear in your mind ......).

The only benefit of using Hadoop is expansion. If your data is a single table with several terabytes of data, full table scan is the strength of Hadoop. In addition, please care for your life and try to stay away from Hadoop. The troubles it brings are not worth mentioning. it saves time and effort using traditional methods.

Note: Hadoop is also a good tool.

I am not mean to have Hadoop. In fact, I often use Hadoop to complete tasks that cannot be easily completed by other tools. (I recommend using Scalding instead of Hive or Pig, because you can use Scala to write cascade Hadoop tasks, hiding the underlying details of MapReduce .) What I want to emphasize in this article is that we should think twice before using Hadoop. don't use 500 MB of data as a mosquito. you should also use a cannon like Hadoop.