Application Master continuous org.apache.hadoop.ipc.Client:Retrying connect to server

first, the problem phenomenon

After a certain nodemanager exits, a large number of the following logs appear in Application master, and it lasts for a long time before application master exits successfully.

2016-06-24 09:32:35,596 info [containerlauncher  #3] org.apache.hadoop.ipc.client:  retrying connect to server: dchadoop206/192.168.1.199:32951. already tried  1 time (s);  retry policy is retryuptomaximumcountwithfixedsleep (maxRetries=10,  sleeptime=1000 milliseconds) 2016-06-24 09:32:35,596 info [containerlauncher  #9]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 1 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  09:32:35,597 info [containerlauncher  #7] org.apache.hadoop.ipc.client: retrying  Connect to server: dchadoop206/192.168.1.199:32951. already tried 1 time (s) ; retry policy is Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  09:32:36,455 info [containerlauncher  #8] org.apache.hadoop.ipc.client: retrying  Connect to server: dchadoop206/192.168.1.199:32951. already tried 7 time (s) ;  retry policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000  milliseconds) 2016-06-24 09:32:36,539 info [containerlauncher  #5]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 4 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  09:32:36,539 info [containerlauncher  #1] org.apache.hadoop.ipc.client: retrying  connect to server: dchadoop206/192.168.1.199: 32951. already tried 4 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  09:32:36,539 info [containerlauncher  #6] org.apache.hadoop.ipc.client: retrying  Connect to server: dchadoop206/192.168.1.199:32951. already tried 3 time (s) ;  retry policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000  milliseconds) 2016-06-24 09:32:36,539 info [containerlauncher  #2]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 3 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  09:32:36,539 info [containerlauncher  #0] ORG.APACHE.HADOOP.IPc. Client: retrying connect to server: dchadoop206/192.168.1.199:32951. already  tried 4 time (s);  retry policy is retryuptomaximumcountwithfixedsleep ( Maxretries=10, sleeptime=1000 milliseconds) 2016-06-24 09:32:36,596 info [ containerlauncher  #4] org.apache.hadoop.ipc.client: retrying connect to server:  dchadoop206/192.168.1.199:32951. already tried 2 time (s);  retry policy  is retryuptomaximumcountwithfixedsleep (Maxretries=10, sleeptime=1000 milliseconds) 2016-06-24 09:32:36,597 info [containerlauncher  #3] org.apache.hadoop.ipc.client:  retrying connect to server: dchadoop206/192.168.1.199:32951. already tried  2 time (s);  retry policy is retryuptomaximumcountwithfixedsleep (maxRetries=10,  sleeptime=1000 milliseconds) 2016-06-24 09:32:36,597 info [containerlauncher  #9] org.apache.hadoop.ipc.client:  retrying connect to server: dchadoop206/192.168.1.199:32951. already tried  2 time (s);  retry policy is retryuptomaximumcountwithfixedsleep (maxRetries=10,  Sleeptime=1000 milliseconds) .... 2016-06-24 12:57:52,328 info [thread-1835] org.apache.hadoop.ipc.client: retrying  connect to server: dchadoop206/192.168.1.199:32951. Already tried 8  Time (s);  retry policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime =1000 milliseconds) 2016-06-24 12:57:53,339 info [thread-1835]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 9 time (s); retry policy is  RetryuptomaximumcountwithfixEdsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24 12:58:04,357 info [ thread-1835] org.apache.hadoop.ipc.client: retrying connect to server:  Dchadoop206/192.168.1.199:32951. already tried 0 time (s);  retry policy is  retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  12:58:05,367 info [thread-1835] org.apache.hadoop.ipc.client: retrying connect  to server: dchadoop206/192.168.1.199:32951. already tried 1 time (s);  Retry policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000  MILLISECONDS) 2016-06-24 12:58:06,378 info [thread-1835] org.apache.hadoop.ipc.client:  retrying connect to server: dchadoop206/192.168.1.199:32951. already tried  2 time (s);  retry pOlicy is retryuptomaximumcountwithfixedsleep (Maxretries=10, sleeptime=1000 milliseconds) 2016-06-24 12:58:07,392 info [thread-1835] org.apache.hadoop.ipc.client: retrying  connect to server: dchadoop206/192.168.1.199:32951. Already tried 3  Time (s);  retry policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime =1000 milliseconds) 2016-06-24 12:58:08,399 info [thread-1835]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 4 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  12:58:09,408 info [thread-1835] org.apache.hadoop.ipc.client: retrying connect  To server: dchadoop206/192.168.1.199:32951. already&nbSp;tried 5 time (s);  retry policy is retryuptomaximumcountwithfixedsleep ( Maxretries=10, sleeptime=1000 milliseconds) 2016-06-24 12:58:10,417 info [thread-1835]  org.apache.hadoop.ipc.client: retrying connect to server: dchadoop206/ 192.168.1.199:32951. already tried 6 time (s); retry policy is  Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 milliseconds) 2016-06-24  12:58:11,425 info [thread-1835] org.apache.hadoop.ipc.client: retrying connect  To server: dchadoop206/192.168.1.199:32951. already tried 7 time (s);  retry  policy is retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000  MILLISECONDS) 2016-06-24 12:58:12,434 info [thread-1835] org.apache.hadoop.ipc.client:  retrying connect to server: dChadoop206/192.168.1.199:32951. already tried 8 time (s);  retry policy is  retryuptomaximumcountwithfixedsleep (Maxretries=10, sleeptime=1000 milliseconds)

1) dchadoop206 on the NodeManager exit (due to restart), causing application master to continue to connect container before NodeManager. Obviously these container are already connected.

2) Finally, after a very long time of about 3-4 hours, the connection is not the exception is thrown, application master normal end.

Second, the problem analysis

This problem mainly involves the RPC mechanism of Hadoop. First of all, see the following two configuration parameters

  #定义client连接到nodemanager的最大超时时间, not a single connection, but after how much time not connected to NodeManager, That the operation failed  <property>        <name> yarn.client.nodemanager-connect.max-wait-ms</name>        < value>15*60*1000</value> </property> #  defines the time interval for each attempt to connect to NodeManager  < Property>        <name> yarn.client.nodemanager-connect.retry-interval-ms</name>         <value>10*1000</value> </property> 

   

Maximumcount:yarn.client.nodemanager-connect.max-wait-ms/yarn.client.nodemanager-connect.retry-interval-ms=90 Times

And each time the RPC request, the client also has its own re-connect policy, is similar to the east:

2016-06-24 09:32:36,455 INFO [containerlauncher #8] org.apache.hadoop.ipc.Client:Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 7 time (s); Retry policy is Retryuptomaximumcountwithfixedsleep (maxretries=10, sleeptime=1000 MILLISECONDS) controlled by two RPC parameters, ipc.client.connect.max.retries=10 and ipc.client.connect.retry.interval=1000ms Control

So finally the wait time for Applicationmaster to discard try Connect is: 90* (10+10) =1800s

Iii. Solutions

1) in the submission of map-reduce/hive sql/hive Server2 and other client machine modification Yarn-site.xml The following parameters
2)Set this parameter via-D in the Hadoop command line

<property> <name>yarn.client.nodemanager-connect.max-wait-ms</name> <value>180000< ;/value> </property>

So the total waiting time is 6 minutes.

Precautions

This modification is not necessary to do any restart Yarn component operation, is a client-related operation!

This article is from the "Scattered People" blog, please be sure to keep this source http://zouqingyun.blog.51cto.com/782246/1881294

Application Master continuous org.apache.hadoop.ipc.Client:Retrying connect to server