Elasticsearch EC2 instance not accessible

bradfordli · September 7, 2016, 8:58pm

I am trying to access my ElasticSearch on a running EC2 instance from outside the Cloud. I currently have SSH/HTTP/HTTPS open to the public for inbound traffic as well as all open for outbound traffic. I set up a public IP for my EC2 instance as well.

By default ElasticSearch is on port 9200. I'm not sure if I configured my elasticsearch.yml file correctly but it basically has the default configuration I only changed the cluster.name to something else.

When I type in my public IP with port 9200 into my local browser or locally do a telnet {public-ip} 9200, there is no response. When I SSH into my EC2 instance. I can perform a curl localhost:9200 and I get the correct response from elasticsearch

How can I connect to my ElasticSearch running on my EC2 instance from outside the cloud?

I added a Custom Rule for my security group for inbound traffic that includes port 9200 and is open to 0.0.0.0/0 and I still cannot access this EC2 instance. My elasticsearch.yml currently has the default values

dadoonet · September 7, 2016, 9:12pm

You need to change network.host

Your instance is currently only listening on 127.0.0.1

bradfordli · September 7, 2016, 11:14pm

@dadoonet Thanks for the reply. I added network.host: {public-ip-ec2}. Now my elasticsearch.yml consists of

cluster.name: trinity-monitoring
node.name: es01
bootstrap.mlockall: true
network.host: "{public-ip-ec2}"

I have the cloud-aws plugin installed but I don't believe I am using any of the features.

but I am now getting the error:

# Exception in thread "main" BindTransportException[Failed to bind to [9300-9400]]; nested: ChannelException[Failed to bind to: /{public-ip-ec2}:9400]; nested: BindException[Cannot assign requested address];
Likely root cause: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:463)
	at sun.nio.ch.Net.bind(Net.java:455)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Refer to the log for complete error details.

I also ran netstat -tulpn to see if anything was also on that port but nothing from 9200-9400 showed up

dadoonet · September 8, 2016, 6:45am

May be try the private IP instead?

bradfordli · September 8, 2016, 5:43pm

@dadoonet

From my understanding, isn't the private IP only accessible from other machines within the VPC?

Anyways, after changing the network.host to my private IP. I can no longer curl localhost:9200 as it gives me curl: (7) Failed to connect to localhost port 9200: Connection refused

However, when I curl {private-ip}:9200 I get the appropriate JSON response.

dadoonet · September 8, 2016, 6:15pm

And then, can you access it from the outside?

bradfordli · September 8, 2016, 6:49pm

@dadoonet nope I cannot. Sorry forgot to mention that part

dadoonet · September 8, 2016, 7:04pm

What gives

ifconfig /all

?

bradfordli · September 8, 2016, 7:12pm

# ifconfig /all
/all: error fetching interface information: Device not found

This is within the EC2 instance

# ifconfig
eth0      Link encap:Ethernet  HWaddr 0A:FD:A5:FD:B5:AB  
          inet addr:10.0.0.198  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::8fd:a5ff:fefd:b5ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:210564 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90149 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:261323576 (249.2 MiB)  TX bytes:10053794 (9.5 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3080 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3080 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:319734 (312.2 KiB)  TX bytes:319734 (312.2 KiB)

dadoonet · September 8, 2016, 7:45pm

So that's the reason you can't bind the public ip.

Not sure how to fix but I guess you need to make somehow change the security group to forward traffic from 9200 public to 9200 private?

aadicodes · January 11, 2017, 1:41am

I had similar issue accessing newly deployed elasticsearch5.1 instance on a newly created ec2 instance.
I added an entry as below in /etc/hosts file. Boom, I was able to curl the es instance with ip and port , ie
curl http://{private-ip-address}:9200/
returns the standard json response.

/etc/hosts entry looks like below
127.0.0.1 localhost 127.0.0.1 {replace with ec2 instance hostname}

also make sure security group is configured with relevant IP and ports.