Jconsole won't connect to remote JVM on Linux

Source: http://bugs.sun.com/bugdatabase/view_bug.do? Bug_id = 6209663

Bug ID:	6209663
Votes	7
Synopsis	Jconsole won't connect to remote JVM on Linux
Category	Java: jconsole
Reported against
Release fixed
State	11-closed, not a defect, Bug
Priority:	3-Medium
Related bugs
Submit date	16-dec-2004
Description	FULL PRODUCT VERSION :java version "1.5.0"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)Java HotSpot(TM) Client VM (build 1.5.0-b64, mixed mode)JDK 1.5.0JDK 1.6.0-EAADDITIONAL OS VERSION INFORMATION :Fedora Core 3 2.6.9-667Debian testing 2.6.8-1Debian unstable 2.6.9-1SUSE ENTERPRISE LINUX 9Windows XP SP 1A DESCRIPTION OF THE PROBLEM :jconsole won't connect to a remote Linux JVM - "connection failed". Tested on different Linux distributions and versions, and on different JVMs.I can connect just fine locally on all Linux platforms, using either Local or Remote tab in jconsole.I can also connect remotely from Linux to JVM on Windows XP, using the same command line and SampleTest.Connecting from Linux to Linux or from Windows to Linux does NOT work, however.I'm not running firewalls or filters on any of these systems. Network is working, cause I'm using ssh/scp/ftp to login and copy files etc. ping between these systems work. I can also connect using telnet remote-ip 8999 from the jconsole box to the remote JVM and see that it actually responds (pressing Enter three times closes the connection).I'm having the same problem for authenticated (still no ssl) connection. This also means that authenticated works *locally*, just not remotely.On the "server" side, I'm using the JMX SampleTest program.(Haven't found anything in the installation notes or release notes indicating this should not work on Linux. Also, haven't found anything in Bug Parade, on java.net/java.sun.com forums or Google.)STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :On the "server" side:java -Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false SampleTest(used -Dcom.sun.management.jmxremote only to test Local-only connection.)On the client side:jconsole remote-ip:8999EXPECTED VERSUS ACTUAL BEHAVIOR :EXPECTED -Connect!ACTUAL -jconsole reports "Connection failed".ERROR MESSAGES/STACK TRACES THAT OCCUR :jconsole reports "Connection failed".REPRODUCIBILITY :This bug can be reproduced often. xxxxx@xxxxx 2004-12-16 19:25:03 GMT
Work around	N/A
Evaluation	I'm able to connect from Windows to Linux and also from Linux to Linux.The application is running on RedHat A.S. 2.1. JDK 5.0 jconsole running on Windows 2000 and RedHat Enterprise Linux 3.0 Update 1 both work!I suspect this is related to the hostname resolution. Suggest the submitter to try entering IP address instead of hostname to verify if it works or not. xxxxx@xxxxx 2004-12-16 20:01:44 GMTSome Linux distributions setup /etc/hosts so that the hostname resolves to the loopback address rather than the host address. If you use "hostname -i" and it reports 127.0.0.1 then you may have problems with RMI and jconsole. Editing /etc/hosts so that the hostname resolves to the host address will fix the issue. xxxxx@xxxxx 2005-2-03 17:20:00 GMTThe submitter has verified that it's a Linux /etc/hosts problem as described above in the evalution. This will be closed as "not a defect". We will document this problem in the JConsole FAQ in the next documentation update under:http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html xxxxx@xxxxx 2005-03-02 19:03:00 GMT

I see a very similar effect. I have one machine running Debian amd64 sarge that I am unable to connect to - I can connect locally using local connection. I can even connect "remotely" on the same machine by using 127.0.0.1 or it's IP address. I just cannot connect remotely from another machine - I get the same "Connection Failed". A netstat shows an ESTABLISHed connection to the correct port, the connection will not go TIME_WAIT until I click cancel on the connection window (even though it says Connection failed). I've tried both Debian i386/amd64 and solaris 10 beta 72 jconsoles (amd64 1.6.0 build) without success. A i386 machine with the exact same java -D args will allow connections. Its very very strange.java version: java version "1.5.0"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)Java HotSpot(TM) 64-Bit Server VM (build 1.5.0-b64, mixed mode)/usr/lib/j2sdk1.5-sun/bin/java -server -Djava.awt.headless=true -Xmx896m -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8400 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath /usr/lib/j2sdk1.5-sun/lib/tools.jar:/usr/lib/j2sdk1.5-sun/jre//lib/jcert.jar:/usr/lib/j2sdk1.5-sun/jre//lib/jnet.jar:/usr/lib/j2sdk1.5-sun/jre//lib/jsse.jar:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar -Dcatalina.base=/var/lib/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/var/lib/tomcat5/temp org.apache.catalina.startup.Bootstrap -config /etc/tomcat5/server0.xml start

Submitted on 21-dec-2004
Bclbob

Futher investigationJava code:public class Sleep{        public static void main( String[] args)        {                try                {                        Thread.sleep(10000000);                }                catch (Exception e)                {                        // eat                }        }}java cmd line:java -Dcom.sun.management.jmxremote.port=8900 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Sleepstill causes the same issue on the amd64 machine, but connects fine on i386.amd64 java -version:java version "1.5.0"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)Java HotSpot(TM) 64-Bit Server VM (build 1.5.0-b64, mixed mode)i386 java -version:java version "1.5.0"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)Java HotSpot(TM) Client VM (build 1.5.0-b64, mixed mode)

Submitted on 29-dec-2004
Opty

Same problem here on Linux Mandrake 10.1 for x86-64 (Xeon EM64T here), when looking on jconsole localy the remote jconsole connection thread is atjava.net.SocketInputStream.socketRead0() for for few seconds before terminating, so the there have been a connection but something failed while communicating...

Submitted on 06-Jan-2005
Bclbob

Still an issue on Linux amd64 with SR1java -versionjava version "1.5.0_01"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08)Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_01-b08, mixed mode)Can't connect remotely from Windows or Linux or Solaris 10 b72.

Submitted on 14-Jan-2005
Masterzen666

I confirm the problem:client on debian/sarge i386server on debian/sarge amd64 (EM64T)Same VM running on client & server (except 64bit one for server):java version "1.5.0_01"Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08)Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_01-b08, mixed mode)Analysis of a network trace seems to show that the remote part sends 127.0.0.1 as IP address. Analysis of a network trace taken on 'lo' (on the client) shows that the jconsole tries to connect to a port that is not open on the client (because it is open on the server).It might be related to the /etc/hosts file and the hostname to IP conversion...I found the culprit and a workaround: my /etc/hosts was:127.0.0.1 localhost localhost.localdomain server4Changed that to 127.0.0.1 localhost localhost.localdomain192.168.0.230 server4fixed the problem....

Submitted on 19-Jan-2005
Bclbob

Thanks masterzen666 - that worked for me! great work

Submitted on 21-Jan-2005
Marcoesch

I dont think it is related to name resolution. I am experiencing  the same problem. On the server I can see the socket going in to LISTEN mode on the specified port. When the client tries to connect I can see it going into ESTABLISHED with ip address from the client running jconsole  ... but still get connection fialed on the client.tcp        0      0 *:9930                  *:*                     LISTENtcp        0      0 ::ffff:10.107.0.15:9930 ::ffff:10.107.0.25:1641 ESTABLISHED

Submitted on 04-feb-2005
Opty

The above evaluation trick for /etc/hosts works with a straight connection, but it won

Submitted on 14-feb-2005
Bjerkz

The "trick" with /etc/hosts solves it for me, too.Thanks!

Submit Ted on 03-Mar-2005
Opty

/etc/hosts doesn

Submit Ted on 03-Mar-2005
Opty

/etc/hosts does not resolve when using NAT or SSH tunneling

Submit Ted on 03-Mar-2005
Caffeine0001

If you are behind a NAT or firewall and having problems, see the second reply of http://forum.java.sun.com/thread.jspa?threadID=289869&messageID=1242288 and http://java.sun.com/j2se/1.4/docs/guide/rmi/faq.html#nethostname.  Please also add this to the JConsole FAQ.

Submitted on 15-feb-2007

I am running Ubuntu Edgy and I definitely see this problem.  I don't know whose fault it is, but it's definitely a bug.  I have a pretty comprehensive writeup including a workaround here:http://trac.marketcetera.org/trac.fcgi/wiki/Marketcetera/UbuntuJMX

Submitted on 19-feb-2007
Holgerk

If have the same problem even with proper hostname setup.JDK 1.5.0.11, Linux/i386, two machines.I can connect to the remote port via "telnet server port", but jconsole reports "connection failed" when started on client (oder server).BTW: jmxremote documentation is inconsistant:1.) gui screwup: I found no place how users and roles relate. all documentation seems to use these terms als synonyms. the remote dialog in jconsole should read "rolename", not "username" in this case.2.) typo: the commed in jmxremote.password reads monitorRole and measureRole.

Submitted on 29-Mar-2007
Sheepshepherd

For my RedHat AS3  the trick with /etc/hosts was not enough until I changed my host name with "hostname -v <name_of_my_machine>"  command (root privileges are required).

Submitted on 26-jul-2007
Mipito

I'm having the same problem on a linux box and the 'hostname' fix doesn't help. Issuing 'hostname -i" gives me the true ip. I fire up the process to be monitored on the server, ssl & authentication disabled. On the client (a pc) i fire up jconsole and the connection is established as displayed by netstat, but for some reason it ends up in "connection failed".What is jconsole trying to do at this point? I can connect fine to that port using telnet.

Submitted on 27-jul-2007
Mipito

Ok, the problem i was running into (see my previous comment) and is now resolved is that the jvm you're trying to connect to actually exposes *two* ports, the one specified via -Dcom.sun.management.jmxremote.port, and some other one. The 2nd one is random, but jconsole wants to connect to it, so if you have a firewall, and you've only opened up the above port, you're hosed. I don't know if it's possible to specify both ports on the cmd line, which is what you'd want to do if you have to poke holes in your firewall.I believe there is a way to write code to get around this problem, but that's a pain, in particular when one is not in control of that code.

Submitted on 22-aug-2008
R_d

adding -Djava.rmi.server.hostname=
Please note: jdk6 is formerly known as Project mustang