Need help setting up ElasticSearch on EC2

Elastic Stack Elasticsearch
1

Hello,

I am following this tutorial to setup ElasticSearch on
EC2: http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html

So far, I finished pretty much every part in the tutorial and I can:

  • Have 2 nodes on EC2 discovering each other and finish master selection
  • request indexing/health check from local machine using curl

The problem is that whenever I try to connect to the Nodes on EC2 by
launching elasticsearch locally (as a client), It times out. I can discover
the nodes on EC2, but I can not connect to them.

Any help/suggestion would be appreciated, I tried Googling and tried
solutions to similar problems without any luck.

Error message and config files below. As shown in the log, the client from
my local machine (S'ym) managed to find the node on EC2 (Zero-G) but can't
connect to it...

[2013-03-15 19:31:52,215][WARN ][discovery.zen ] [S'ym] failed
to connect to master
[[Zero-G][H7rF8s6IT_-R1oTjaS2Epw][inet[/10.168.34.109:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException:
[Zero-G][inet[/10.168.34.109:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:665)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:604)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:574)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:127)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(ZenDiscovery.java:336)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:289)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: connection timed out
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:136)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

Config on EC2:
cluster.name:
cloud:
aws:
access_key:
secret_key:
region: us-east-1
discovery:
type: ec2
ec2:
groups: ElasticSearch
path:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch

Config on my local machine:

cluster.name:
cloud:
aws:
access_key:
secret_key:
region: us-east-1
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts:
["ec2-XXX-XX-XXX-XX.compute-1.amazonaws.com"]
client: true
data: false

Thanks!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Did you open your EC2 ports (9300)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 mars 2013 à 03:39, Fish Tastic whoisbingo@gmail.com a écrit :

Hello,

I am following this tutorial to setup ElasticSearch on EC2: http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html

So far, I finished pretty much every part in the tutorial and I can:

  • Have 2 nodes on EC2 discovering each other and finish master selection
  • request indexing/health check from local machine using curl

The problem is that whenever I try to connect to the Nodes on EC2 by launching elasticsearch locally (as a client), It times out. I can discover the nodes on EC2, but I can not connect to them.

Any help/suggestion would be appreciated, I tried Googling and tried solutions to similar problems without any luck.

Error message and config files below. As shown in the log, the client from my local machine (S'ym) managed to find the node on EC2 (Zero-G) but can't connect to it...

[2013-03-15 19:31:52,215][WARN ][discovery.zen ] [S'ym] failed to connect to master [[Zero-G][H7rF8s6IT_-R1oTjaS2Epw][inet[/10.168.34.109:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Zero-G][inet[/10.168.34.109:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:665)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:604)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:574)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:127)
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(ZenDiscovery.java:336)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: connection timed out
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:136)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

Config on EC2:
cluster.name:
cloud:
aws:
access_key:
secret_key:
region: us-east-1
discovery:
type: ec2
ec2:
groups: ElasticSearch
path:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch

Config on my local machine:

cluster.name:
cloud:
aws:
access_key:
secret_key:
region: us-east-1
discovery.zen.ping.unicast.enabled: true
discovery.zen.ping.unicast.hosts: ["ec2-XXX-XX-XXX-XX.compute-1.amazonaws.com"]
client: true
data: false

Thanks!!

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks for the reply.

On AWS security group, I opened all ports to my ip address for UDP, TCP and
ICMP. So I don't think it's an issue with port.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Did you set host_type to public_ip?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 mars 2013 à 04:17, Fish Tastic whoisbingo@gmail.com a écrit :

Thanks for the reply.

On AWS security group, I opened all ports to my ip address for UDP, TCP and ICMP. So I don't think it's an issue with port.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks for the reply.

I added "host_type: public_ip" to AWS node and I am stilling getting the
same error. When the local client tries to connect to the node on EC2, it
just times out.

I am pretty sure it's a configuration issue but I can't figure it out.

full log from local client below

[2013-03-18 14:13:57,863][INFO ][node ] [Apollo]
{0.20.5}[78553]: initializing ...
[2013-03-18 14:13:57,879][INFO ][plugins ] [Apollo] loaded
[cloud-aws], sites []
[2013-03-18 14:13:59,493][DEBUG][discovery.zen.ping.multicast] [Apollo]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2013-03-18 14:13:59,498][DEBUG][discovery.zen.ping.unicast] [Apollo] using
initial hosts [], with concurrent_connects [10]
[2013-03-18 14:13:59,500][DEBUG][discovery.ec2 ] [Apollo] using
ping.timeout [5s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-03-18 14:13:59,501][DEBUG][discovery.zen.elect ] [Apollo] using
minimum_master_nodes [-1]
[2013-03-18 14:13:59,502][DEBUG][discovery.zen.fd ] [Apollo]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-03-18 14:13:59,507][DEBUG][discovery.zen.fd ] [Apollo] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-03-18 14:13:59,852][DEBUG][discovery.ec2 ] [Apollo] using
host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true],
availability_zones [[]]
[2013-03-18 14:14:00,999][INFO ][node ] [Apollo]
{0.20.5}[78553]: initialized
[2013-03-18 14:14:00,999][INFO ][node ] [Apollo]
{0.20.5}[78553]: starting ...
[2013-03-18 14:14:01,151][INFO ][transport ] [Apollo]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.1.10.153:9300]}
[2013-03-18 14:14:01,156][TRACE][discovery ] [Apollo]
waiting for 30s for the initial state to be set by the discovery
[2013-03-18 14:14:01,818][TRACE][discovery.ec2 ] [Apollo]
building dynamic unicast discovery nodes...
[2013-03-18 14:14:01,824][TRACE][discovery.ec2 ] [Apollo] adding
i-f296c5c0, address 10.253.71.69, transport_address inet[/10.253.71.69:9300]
[2013-03-18 14:14:01,824][TRACE][discovery.ec2 ] [Apollo] not
adding i-ce8bd8fc, state stopped is not pending or running
[2013-03-18 14:14:01,824][DEBUG][discovery.ec2 ] [Apollo] using
dynamic discovery nodes [[#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]]
[2013-03-18 14:14:01,825][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
connecting (light) to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
[2013-03-18 14:14:04,467][TRACE][discovery.ec2 ] [Apollo]
building dynamic unicast discovery nodes...
[2013-03-18 14:14:04,468][TRACE][discovery.ec2 ] [Apollo] adding
i-f296c5c0, address 10.253.71.69, transport_address inet[/10.253.71.69:9300]
[2013-03-18 14:14:04,468][TRACE][discovery.ec2 ] [Apollo] not
adding i-ce8bd8fc, state stopped is not pending or running
[2013-03-18 14:14:04,468][DEBUG][discovery.ec2 ] [Apollo] using
dynamic discovery nodes [[#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]]
[2013-03-18 14:14:04,471][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
connecting (light) to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
[2013-03-18 14:14:07,121][TRACE][discovery.ec2 ] [Apollo]
building dynamic unicast discovery nodes...
[2013-03-18 14:14:07,122][TRACE][discovery.ec2 ] [Apollo] adding
i-f296c5c0, address 10.253.71.69, transport_address inet[/10.253.71.69:9300]
[2013-03-18 14:14:07,123][TRACE][discovery.ec2 ] [Apollo] not
adding i-ce8bd8fc, state stopped is not pending or running
[2013-03-18 14:14:07,123][DEBUG][discovery.ec2 ] [Apollo] using
dynamic discovery nodes [[#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]]
[2013-03-18 14:14:07,123][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
connecting (light) to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
[2013-03-18 14:14:09,625][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
disconnecting from [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
[2013-03-18 14:14:31,158][WARN ][discovery ] [Apollo] waited
for 30s and no initial state was set by the discovery
[2013-03-18 14:14:31,158][INFO ][discovery ] [Apollo]
xiao-elasticsearch/5KmNGgEZT0ad2SGWmjL3EA
[2013-03-18 14:14:31,163][INFO ][http ] [Apollo]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.1.10.153:9200]}
[2013-03-18 14:14:31,163][INFO ][node ] [Apollo]
{0.20.5}[78553]: started
[2013-03-18 14:14:32,342][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
failed to connect to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[/10.253.71.69:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:638)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:600)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:569)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:131)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:273)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: connection timed out
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:136)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2013-03-18 14:14:32,346][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
failed to connect to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[/10.253.71.69:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:638)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:600)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:569)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:131)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:273)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:106)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:69)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:570)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at
org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:635)
... 7 more
[2013-03-18 14:14:32,342][TRACE][discovery.ec2 ] [Apollo] full
ping responses: {none}
[2013-03-18 14:14:32,350][DEBUG][discovery.ec2 ] [Apollo]
filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-03-18 14:14:32,349][TRACE][discovery.zen.ping.unicast] [Apollo] [1]
failed to connect to [#cloud-i-f296c5c0-0][inet[/10.253.71.69:9300]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[/10.253.71.69:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:638)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:600)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:569)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:131)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:273)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:106)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:69)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:570)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:634)
at
org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:635)
... 7 more
[2013-03-18 14:14:32,353][INFO ][cluster.service ] [Apollo]
new_master [Apollo][5KmNGgEZT0ad2SGWmjL3EA][inet[/10.1.10.153:9300]],
reason: zen-disco-join (elected_as_master)
[2013-03-18 14:14:32,478][INFO ][gateway ] [Apollo]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

I think I was doing the wrong thing.

I tried to test my EC2 cluster by creating a local node to join the
cluster, this didn't work because the nodes are in different regions.

To start indexing, should I go straight to creating a TransportClient in
java and try to connect to my node?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Definitely yes!

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 19 mars 2013 à 00:14, Fish Tastic whoisbingo@gmail.com a écrit :

I think I was doing the wrong thing.

I tried to test my EC2 cluster by creating a local node to join the cluster, this didn't work because the nodes are in different regions.

To start indexing, should I go straight to creating a TransportClient in java and try to connect to my node?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.