Configuring elasticsearch 6.X cluster on aws instances

Elastic Stack Elasticsearch
1

Hi,

I am trying to set up elasticsearch on my aws instance. I have installed elasticsearch 6.6 on the instance and elasticsearch is running locally. What configuration changes should I make in the elasticsearch.yml file to make me access the elasticsearch data using the instance's public IP address.

ex: I have made ssh connection to the instance and run the command
curl http://localhost:9200

This gives me the correct output on the console. But I want to access the same by the public IP address of the instance.
http://public.ip.address:9200

When I try this on the browser, nothing happens(session time out after some time)

I was looking at a tutorial here but found that it is for elasticsearch 4.X and things have changed in elasticsearch 6.X.

I tried configuring the elasticsearch.yml file using this tutorial and have come this far:

cluster.name: elasticsearch-demo
node.name: elasticsearch-node-1
node.master: true
node.data: true
node.ingest: true
discovery.zen.hosts_provider: ec2
discovery.zen.ping.unicast.hosts: [private ip]
network.host: [private ip]

What configuration settings should I change to get this running.

Or is there any link to a tutorial on how to manually setup elasticsearch 6.X on aws.

Thank you

2

Please don't expose an unsecured Elasticsearch node on a public IP address. It means that anyone in the world can read your data, modify it, wipe it out, use the cluster for storing malware, and all sorts of other things. There are malicious actors that scan for such Elasticsearch installations and do these things.

If you want a quick-and-dirty solution then you can use a SSH tunnel to expose the node's HTTP port (9200) on your local node. The proper solution is to use the security functionality and talk to your node over HTTPS.

1 Like
3

Hi @DavidTurner, I tried using SSH tunnel by forwarding 9200 port of my local machine to the remote elasticsearch server but I get operation time out error.

I changed my elasticsearch.yml file to have this:

network.host: 0.0.0.0
http.port: 9200

I used the tunnel command:

ssh -i <identity_file> user@<bastion_machine_ip> -L 9200:<remote_server's_private_ip>:9200 -N -v

I get this error on the SSH terminal:

channel 2: open failed: connect failed: Connection timed out
debug1: channel 2: free: direct-tcpip: listening port 9200 for <remote_server's_private_ip> port 9200, connect from ::1 port 52303 to ::1 port 9200, nchannels 3
4

I don't know, really. Perhaps you could try and set network.host to the actual IP address? The SSH log messages talk about ::1 which is the IPv6 address for localhost, and I don't know that IPv6 should be in play here.

1 Like
5

setting network.host to the actual IP address still works on the server but from the remote machine when I try http://<actual_ip>:9200 causes timeout.

http://localhost:9200 with port forwarding enabled makes the request to be refused by the server now.

6

Yes that makes sense, you're forwarding your local port 9200 to the remote machine, so you should be trying to talk to your local port.

Hmm. How about setting network.host: 127.0.0.1 and then calling ssh with -L 9200:127.0.0.1:9200?

7

Hi @DavidTurner, this doesn't work. I still get the timeout error. Also, I am not connecting to the elasticsearch server directly. I first connect to a bastion server. Only this server has direct access to the private IP of the elasticsearch server.

So, I think tunneling command should have 9200:remote_server's_private_ip:9200.

I am using this command for port forwarding:

ssh -i <identity_file> user@<bastion_machine_ip> -L 9200:<remote_server's_private_ip>:9200 -N -v

8

In which case I think you will need to set network.host: remote_server's_private_ip.

9

Hi @DavidTurner, it is still not solved after setting network.host: remote_server's_private_ip

I tried curl -X GET "10.76.186.211:9200/index_name" from the bastion machine and also my local machine after port forwarding. Server timed out for both requests.

10

Can you otherwise communicate between the bastion machine and the Elasticsearch server on that address? Can you share the output of netstat -ant run on the Elasticsearch instance?

11

Yes, I log into the elasticsearch server from the bastion machine itself.

Here is the output you want:

31%20PM
Screen Shot 2019-02-15 at 2.05.31 PM.jpg1132×348 115 KB

IP behind red is the private IP of the elasticsearch server and the one behind the blue is the bastion machine IP.

12

The Proto column says tcp6 which suggests to me that it's only listening for IPv6 connections. Yet the address looks like an IPv4 address. I'm baffled.

13

Hmm, ok, this is apparently legitimate. Do your security groups allow a connection between the bastion host and the Elasticsearch node on port 9200?

14

Hi @David,

It was a configuration issue at my end. I had set inbound rules to enable requests at port 9200 on the elasticsearch server but the outbound rules on the bastion server were not updated for the same.

I made the changes and it worked.

Thanks for your help on this :grinning:

1 Like
15

You're welcome @amitnegi6190, we got there in the end :slight_smile:

1 Like
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.