Hive error when executing query statement

Hive Error when executing query statement: org.apache.hadoop.ipc.RemoteException:java.io.IOException:java.io.IOException:

Hive> Select product_id, track_time from Trackinfo limit 5;  
Total MapReduce jobs = 1 Launching Job 1 out of 1 number of reduce tasks are set to 0 since there ' s no reduce operator  Org.apache.hadoop.ipc.RemoteException:java.io.IOException:java.io.IOException:The number of tasks for this job 156028  
        exceeds the configured limit 5000 at Org.apache.hadoop.mapred.JobTracker.submitJob (jobtracker.java:3943) At Sun.reflect.GeneratedMethodAccessor17.invoke (Unknown Source) at Sun.reflect.DelegatingMethodAccessorImpl. Invoke (DELEGATINGMETHODACCESSORIMPL.JAVA:25) at Java.lang.reflect.Method.invoke (method.java:597) at or G.apache.hadoop.ipc.rpc$server.call (rpc.java:563) at Org.apache.hadoop.ipc.server$handler$1.run (Server.java:1388 ) at Org.apache.hadoop.ipc.server$handler$1.run (server.java:1384) at java.security.AccessController.do Privileged (Native method) at JAVAX.SECURITY.AUTH.SUbject.doas (subject.java:396) at Org.apache.hadoop.security.UserGroupInformation.doAs (Usergroupinformation.java:  1093) at Org.apache.hadoop.ipc.server$handler.run (server.java:1382) caused by:java.io.IOException:The number The tasks for this job 156028 exceeds the configured limit 5000 at Org.apache.hadoop.mapred.JobInProgress.checkT   
        Asklimits (jobinprogress.java:509) at org.apache.hadoop.mapred.jobinprogress.<init> (JobInProgress.java:485) At Org.apache.hadoop.mapred.JobTracker.submitJob (jobtracker.java:3941) ... Org.apache.hadoop.ipc.Client.call at ORG.APACHE.HADOOP.IPC.RPC$INV (client.java:1066) Oker.invoke (rpc.java:225) at org.apache.hadoop.mapred. $Proxy 11.submitJob (Unknown Source) at Org.apache . Hadoop.mapred.jobclient$2.run (jobclient.java:921) at Org.apache.hadoop.mapred.jobclient$2.run (JobClient.java:85 0) at Java.security.AccessController.doPRivileged (Native method) at Javax.security.auth.Subject.doAs (subject.java:396) at org.apache.hadoop.se Curity. Usergroupinformation.doas (usergroupinformation.java:1093) at Org.apache.hadoop.mapred.JobClient.submitJobInterna L (jobclient.java:850) at the Org.apache.hadoop.mapred.JobClient.submitJob (jobclient.java:824) at Org.apach E.hadoop.hive.ql.exec.execdriver.execute (execdriver.java:447) at ORG.APACHE.HADOOP.HIVE.QL.EXEC.MAPREDTASK.EXECU Te (mapredtask.java:136) at the Org.apache.hadoop.hive.ql.exec.Task.executeTask (task.java:138) at Org.apach E.hadoop.hive.ql.exec.taskrunner.runsequential (taskrunner.java:57) at Org.apache.hadoop.hive.ql.Driver.launchTas K (driver.java:1336) at the Org.apache.hadoop.hive.ql.Driver.execute (driver.java:1122) at Org.apache.hadoop   
        . Hive.ql.Driver.run (driver.java:935) at ORG.APACHE.HADOOP.HIVE.CLI.CLIDRIVER.PROCESSLOCALCMD (clidriver.java:259) at org. APACHE.HADOOP.HIVE.CLI.CLIDRIVER.PROCESSCMD (clidriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.process Line (clidriver.java:412) at Org.apache.hadoop.hive.cli.CliDriver.run (clidriver.java:755) at Org.apache . Hadoop.hive.cli.CliDriver.main (clidriver.java:613) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native metho d) at Sun.reflect.NativeMethodAccessorImpl.invoke (nativemethodaccessorimpl.java:39) at Sun.reflect.Del Egatingmethodaccessorimpl.invoke (delegatingmethodaccessorimpl.java:25) at Java.lang.reflect.Method.invoke . java:597) at Org.apache.hadoop.util.RunJar.main (runjar.java:156) Job submission failed with exception ' Org.ap Ache.hadoop.ipc.RemoteException (java.io.IOException:java.io.IOException:The number of tasks for this job 156028  
        exceeds the configured limit 5000 at Org.apache.hadoop.mapred.JobTracker.submitJob (jobtracker.java:3943) At Sun.reflect.GeneratedMethoDaccessor17.invoke (Unknown Source) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Delegatingmethodaccessorim PL.JAVA:25) at Java.lang.reflect.Method.invoke (method.java:597) at org.apache.hadoop.ipc.rpc$server.ca LL (rpc.java:563) at the Org.apache.hadoop.ipc.server$handler$1.run (server.java:1388) at Org.apache.hadoop. Ipc. Server$handler$1.run (server.java:1384) at Java.security.AccessController.doPrivileged (Native method) a T Javax.security.auth.Subject.doAs (subject.java:396) at Org.apache.hadoop.security.UserGroupInformation.doAs (use rgroupinformation.java:1093) at Org.apache.hadoop.ipc.server$handler.run (server.java:1382) Caused By:java.io. Ioexception:the number of tasks for this job 156028 exceeds the configured limit 5000 at ORG.APACHE.HADOOP.MAPR Ed. Jobinprogress.checktasklimits (jobinprogress.java:509) at org.apache.hadoop.mapred.jobinprogress.<init> (Job Inprogress.java:4) at Org.apache.hadoop.mapred.JobTracker.submitJob (jobtracker.java:3941) ... More] ' failed:execution Error, return code 1 from Org.apache.hadoop.hive.ql.exec.MapRedTask

Error Reason:

Because the data in the Trackinfo table is too large, and the written SQL statement

Select product_id, track_time from Trackinfo limit 5

The sexual energy loss is too large, which will result in too many map numbers scheduled, exceeding the job limit.

Look at the amount of data in the Trackinfo table in hive:

-bash-3.2$ Hadoop fs-dus/data/share/trackinfo  
Warning: $HADOOP _home is deprecated.  
      
Hdfs://yhd-hadoop06.int.yihaodian.com:9000/data/share/trackinfo 19387740988708

As you can see, the amount of data in Trackinfo is about 19TB so large

Overwrite SQL statement, specify where condition, improve performance

Select product_id, track_time from Trackinfo where ds= ' 2014-5-13 ' limit 5

The problem was solved at the same time.

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/database/extra/