Amazon EC2 instances do not appear in Mathworks Admin Center

I am trying to understand how Matlab parallel computing works with multiple "instances" (separate nodes) in the Amazon EC2 environment. So far, I have figured out almost all of the network, permissions, and security issues with Amazon EC2 in order to get Matlab to talk to individual nodes. I can Add a single EC2 node in the Admin Center "hosts" panel. I can see that "reachable" is green and MDCE Service Status is green.

However, I am running into an odd problem now. I started a second EC2 instance as another node, using the same exact AMI setup as the first node. If you are unfamiliar with Amazon EC2, the AMI is essentially a disc image of the boot drive. It contains the operating system and anything that you installed into the AMI. In my case, I installed Matlab into the AMI. So any instance I boot with the same AMI will have Matlab installed.

Now, when I attempt to add the new node to the hosts panel in Admin Center, I receive no errors. But, the new node does not show up in the hosts panel! I run a "Test Connectivity", and everything looks reasonable (there's a few hostname resolution infos, but nothing major). Next, just to see if I can reach the second node on it's own, I start a "New Session" in Admin Center to flush out all the information of the old hosts. I add the second node, and this time it shows up in the Admin Center hosts panel! But when I run the "Test Connectivity", I now get failures in the hostname consistency.

I've attached an image below that shows what is going on. And then below that, I've pasted the logs from the first node Test Connectivity and from the second node Test Connectivity.

Screen shots of the Admin Center in action:

http://i.imgur.com/BoHF3.png

Log output from -138 EC2 instance.

ID #  Start Time  Stop Time  Executing Node  Subject Node  Test Category  Test Description  Result  Message  
1  Dec 3, 2012 3:18:56 AM  Dec 3, 2012 3:18:56 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  CLIENT_HOSTNAMES  ResolveHostname Test  SUCCESS    
2  Dec 3, 2012 3:18:56 AM  Dec 3, 2012 3:18:56 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  GetClientInfo Test  SUCCESS    
3  Dec 3, 2012 3:18:56 AM  Dec 3, 2012 3:18:56 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  OpenServerSocket Test (on port 27371+)  SUCCESS    
4  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  ResolveHostname Test  SUCCESS    
5  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  CLIENT_HOSTNAMES  ResolveIPToHostname Test  INFO  Hostname (ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com) and canonical hostname (XXX.XXX.XXX.177) do not match.  
6  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MDCE_CONNECT  LockDown Test  SUCCESS    
7  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (XXX.XXX.XXX.177) do not match.  
8  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MDCE_HOSTNAME  GetServerInfo Test  SUCCESS    
9  Dec 3, 2012 3:18:57 AM  Dec 3, 2012 3:18:57 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ResolveHostname Test  SUCCESS    
10  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:58 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MDCE_HOSTNAME  ResolveHostname Test  SUCCESS    
11  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:58 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (XXX.XXX.XXX.177) do not match.  
12  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:58 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MPI_HOSTNAME  ResolveHostname Test  SUCCESS    
13  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:58 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MDCE_HOSTNAME  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (XXX.XXX.XXX.177) do not match.  
14  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:59 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  PingServerSocketHost Test  SUCCESS    
15  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:59 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ConnectToServerSocket Test (remote port 27371)  SUCCESS    
16  Dec 3, 2012 3:18:58 AM  Dec 3, 2012 3:18:59 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MPI_HOSTNAME  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (XXX.XXX.XXX.177) do not match.  
17  Dec 3, 2012 3:18:59 AM  Dec 3, 2012 3:18:59 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  PORTS_AVAILABLE  CheckServices Test (on port 27350)  SUCCESS    
18  Dec 3, 2012 3:19:00 AM  Dec 3, 2012 3:19:00 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  PORTS_AVAILABLE  OpenServerSocket Test (on port 27355+)  SUCCESS    
19  Dec 3, 2012 3:19:00 AM  Dec 3, 2012 3:19:01 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  Cleanup Test  SUCCESS    
20  Dec 3, 2012 3:19:00 AM  Dec 3, 2012 3:19:01 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  PORT_CONNECT  PingServerSocketHost Test  SUCCESS    
21  Dec 3, 2012 3:19:00 AM  Dec 3, 2012 3:19:01 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  PORT_CONNECT  ConnectToServerSocket Test (remote port 27355)  SUCCESS    
22  Dec 3, 2012 3:19:01 AM  Dec 3, 2012 3:19:02 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  PORTS_AVAILABLE  Cleanup Test  SUCCESS    
23  Dec 3, 2012 3:19:02 AM  Dec 3, 2012 3:19:02 AM  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-138.compute-1.amazonaws.com  MDCE_CONNECT  Cleanup Test  SUCCESS

Log output from -81 EC2 instance.

ID #  Start Time  Stop Time  Executing Node  Subject Node  Test Category  Test Description  Result  Message  
1  Dec 3, 2012 3:21:20 AM  Dec 3, 2012 3:21:20 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  CLIENT_HOSTNAMES  ResolveHostname Test  SUCCESS    
2  Dec 3, 2012 3:21:20 AM  Dec 3, 2012 3:21:20 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  GetClientInfo Test  SUCCESS    
3  Dec 3, 2012 3:21:20 AM  Dec 3, 2012 3:21:20 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  OpenServerSocket Test (on port 27371+)  SUCCESS    
4  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  ResolveHostname Test  SUCCESS    
5  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  CLIENT_HOSTNAMES  ResolveIPToHostname Test  INFO  Hostname (ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com) and canonical hostname (domu-12-XXX-XXX-XXX-31-eb.compute-1.internal) do not match.  
6  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MDCE_CONNECT  LockDown Test  SUCCESS    
7  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (XXX.XXX.XXX.177) do not match.  
8  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MDCE_HOSTNAME  GetServerInfo Test  SUCCESS    
9  Dec 3, 2012 3:21:21 AM  Dec 3, 2012 3:21:21 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ResolveHostname Test  SUCCESS    
10  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MDCE_HOSTNAME  ResolveHostname Test  SUCCESS    
11  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ResolveIPToHostname Test  ERROR  com.mathworks.toolbox.distcomp.admincenter.testing.infra.util.TestFailureException: No address in list [/XXX.XXX.XXX.177, /0:0:0:0:0:0:0:1, /127.0.0.1, /XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:a24e%10, /XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:5db1%12, /2001:0:XXXX:XXXX:XXXX:XXXX:XXXX:a24e, /XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:2757%11] matched EC2AMAZ-XXXXXX1's address EC2AMAZ-XXXXXX1/10.XXX.XXX.25.  
12  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MPI_HOSTNAME  ResolveHostname Test  SUCCESS    
13  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  PingServerSocketHost Test  SKIPPED  previous test failed: ResolveIPToHostname Test (POOL2CLIENT) (ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com -> client): ERROR  
14  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2amaz-XXXXXX1  POOL2CLIENT  ConnectToServerSocket Test  SKIPPED  previous test failed: ResolveIPToHostname Test (POOL2CLIENT) (ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com -> client): ERROR  
15  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MDCE_HOSTNAME  ResolveIPToHostname Test  ERROR  com.mathworks.toolbox.distcomp.admincenter.testing.infra.util.TestFailureException: No address in list [/0:0:0:0:0:0:0:1, /127.0.0.1, /fe80:0:0:0:XXXX:XXXX:XXXX:cde6%10, /2001:0:4137:XXXX:XXXX:XXXX:XXXX:cde6, /fe80:0:0:XXXX:XXXX:XXXX:XXXX:3219%12, /fe80:0:0:0:XXXX:XXXX:XXXX:317c%11, /10.XXX.XXX.25] matched EC2AMAZ-XXXXXX1's address EC2AMAZ-XXXXXX1/XXX.XXX.XXX.177.  
16  Dec 3, 2012 3:21:22 AM  Dec 3, 2012 3:21:22 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MPI_HOSTNAME  ResolveIPToHostname Test  INFO  Hostname (ec2amaz-XXXXXX1) and canonical hostname (10.XXX.XXX.25) do not match.  
17  Dec 3, 2012 3:21:23 AM  Dec 3, 2012 3:21:23 AM  ec2amaz-XXXXXX1  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  PORTS_AVAILABLE  CheckServices Test  SKIPPED  previous test failed: ResolveIPToHostname Test (MDCE_HOSTNAME) (client -> ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com): ERROR  
18  Dec 3, 2012 3:21:23 AM  Dec 3, 2012 3:21:23 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  PORTS_AVAILABLE  OpenServerSocket Test  SKIPPED  previous test failed: ResolveIPToHostname Test (MDCE_HOSTNAME) (client -> ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com): ERROR  
19  Dec 3, 2012 3:21:23 AM  Dec 3, 2012 3:21:23 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  PORT_CONNECT  ConnectToServerSocket Test  SKIPPED  previous test failed: OpenServerSocket Test (PORTS_AVAILABLE) (ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com -> ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com): SKIPPED  
20  Dec 3, 2012 3:21:23 AM  Dec 3, 2012 3:21:24 AM  ec2amaz-XXXXXX1  ec2amaz-XXXXXX1  CLIENT_TEST  Cleanup Test  SUCCESS    
21  Dec 3, 2012 3:21:24 AM  Dec 3, 2012 3:21:24 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  PORTS_AVAILABLE  Cleanup Test  SKIPPED  previous test failed: OpenServerSocket Test (on port 27355+) (PORTS_AVAILABLE) (ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com -> ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com): SKIPPED  
22  Dec 3, 2012 3:21:23 AM  Dec 3, 2012 3:21:24 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  PORT_CONNECT  PingServerSocketHost Test  SUCCESS    
23  Dec 3, 2012 3:21:24 AM  Dec 3, 2012 3:21:24 AM  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  ec2-XXX-XXX-XXX-81.compute-1.amazonaws.com  MDCE_CONNECT  Cleanup Test  SUCCESS

I am guessing this is some kind of issue specific to using virtual cloud servers like in Amazon EC2. I had two actual physically separate hosts in my local network to learn / test the Admin Center, and they both showed up separately every time. This issue only started happening when I moved to Amazon EC2 virtual servers. So far, I have yet to ever have 2 separate virtual Amazon EC2 servers show up together in the Admin Center with reachable = green and MDCE Service Status = green. I was able to accomplish this with my local hosts on my local network. The two Amazon EC2 servers are exactly the same Matlab installation, same AMI, same available zone, same virtual machine type, same everything.

It almost feels to me like Admin Center is being confused by the existence of these "virtual" servers. Somewhere in the process of resolving the public IP, local IP, public DNS, or local DNS, Admin Center thinks that these virtual servers are the same physical servers. Therefore, Admin Center does not account for them separate. Instead, updating its LOG files, configurations, or something to essentially say that the same server can be found at 2 separate public IPs or public DNSs.

Has anyone tinkered with Amazon EC2 instances in this manner? Any ideas?