Breaking up ELK stack to individual machines

Elastic Stack Elasticsearch
1

Hello there,

I am trying to break apart my current install of ELK that lives on one machine to an individual machine for Logstash, ElasticSearch, and Kibana. It seems as if connections to ES are being blocked as neither my Logstash or Kibana machine can connect to ES. I'm using IP and the default port (9200). I've set "network.host" to the IP of the ES node in the elasticsearch.yml file. If I understand correctly this is how you configure ES to be accessible on the address set in "network.host" to external machines, yet this doesn't seem to work. I'm not running any firewalls and if I put Kibana or Logstash on the same machine with ES installled it works fine. Any help would be greatly appreciated, thanks!

2

Can you curl to the ES host from a remote machine to see if it's accessible?

3

I get connection refused.

4

Can you post the first few log lines from when ES starts up?

5 
[2015-12-22 09:19:46,896][INFO ][node                     ] [SEA-2670-42.paraport.com] stopping ...
[2015-12-22 09:19:46,971][INFO ][node                     ] [SEA-2670-42.paraport.com] stopped
[2015-12-22 09:19:46,971][INFO ][node                     ] [SEA-2670-42.paraport.com] closing ...
[2015-12-22 09:19:46,988][INFO ][node                     ] [SEA-2670-42.paraport.com] closed
[2015-12-22 09:20:38,945][INFO ][node                     ] [SEA-2670-42.paraport.com] version[1.7.1], pid[6014], build[b88f43f/2015-07-29T09:54:16Z]
[2015-12-22 09:20:38,945][INFO ][node                     ] [SEA-2670-42.paraport.com] initializing ...
[2015-12-22 09:20:39,056][INFO ][plugins                  ] [SEA-2670-42.paraport.com] loaded [], sites []
[2015-12-22 09:20:39,115][INFO ][env                      ] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/SEA--2670--87--vg-root)]], net usable_space [33.9gb], net total_space [38gb], types [ext4]
[2015-12-22 09:20:42,310][WARN ][script                   ] [SEA-2670-42.paraport.com] deprecated setting [script.disable_dynamic] is set, replace with fine-grained scripting settings (e.g. script.inline, script.indexed, script.file)
[2015-12-22 09:20:42,705][INFO ][node                     ] [SEA-2670-42.paraport.com] initialized
[2015-12-22 09:20:42,706][INFO ][node                     ] [SEA-2670-42.paraport.com] starting ...
[2015-12-22 09:20:42,790][INFO ][transport                ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[localhost/127.0.0.1:9300]}
[2015-12-22 09:20:42,803][INFO ][discovery                ] [SEA-2670-42.paraport.com] elasticsearch/Isb20mFoRDWWGtGSxlyMIQ
[2015-12-22 09:20:45,829][INFO ][cluster.service          ] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][Isb20mFoRDWWGtGSxlyMIQ][SEA-2670-42.paraport.com][inet[localhost/127.0.0.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-22 09:20:45,852][INFO ][http                     ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2015-12-22 09:20:45,853][INFO ][node                     ] [SEA-2670-42.paraport.com] started
[2015-12-22 09:20:45,883][INFO ][gateway                  ] [SEA-2670-42.paraport.com] recovered [1] indices into cluster_state

I looked at these logs before posting and found it odd that it didn't seem to reference the network.host config I changed, instead it just looks like its publishing on localhost. That being said, I don't really know how to interpret these logs. Thank you for your help!

6

It's still only listening on localhost, what is the config you have set?

7

Everything in the config is commented out except for this line

network.host: 10.41.150.249

which is the IP of the machine. The documentation reads as if this is the only thing that needs to be set.

8

Try network.host: 0.0.0.0 and restart it, you should see that inet part of the log change.

9

I'm getting errors in the logs now after changing that. I'll need to take another look at this with fresh eyes in the morning. I might just rebuild the machine from scratch. Thanks again for your help! I'll report back here if I find the solution or need any additional assistance.

10

Alright so I'm still getting connection refused when trying to curl but the logs at least look like its publishing the hostname. I tried curling 9200 and 9300

[2015-12-23 11:47:07,518][INFO ][node                     ] [SEA-2670-42.paraport.com] stopping ...
[2015-12-23 11:47:07,562][INFO ][node                     ] [SEA-2670-42.paraport.com] stopped
[2015-12-23 11:47:07,562][INFO ][node                     ] [SEA-2670-42.paraport.com] closing ...
[2015-12-23 11:47:07,569][INFO ][node                     ] [SEA-2670-42.paraport.com] closed
[2015-12-23 11:47:32,803][INFO ][node                     ] [SEA-2670-42.paraport.com] version[1.7.1], pid[9869], build[b88f43f/2015-07-29T09:54:16Z]
[2015-12-23 11:47:32,803][INFO ][node                     ] [SEA-2670-42.paraport.com] initializing ...
[2015-12-23 11:47:32,907][INFO ][plugins                  ] [SEA-2670-42.paraport.com] loaded [], sites []
[2015-12-23 11:47:32,959][INFO ][env                      ] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/SEA--2670--87--vg-root)]], net usable_space [33.5gb], net total_space [38gb], types [ext4]
[2015-12-23 11:47:36,138][INFO ][node                     ] [SEA-2670-42.paraport.com] initialized
[2015-12-23 11:47:36,139][INFO ][node                     ] [SEA-2670-42.paraport.com] starting ...
[2015-12-23 11:47:36,224][INFO ][transport                ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9300]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9300]}
[2015-12-23 11:47:36,238][INFO ][discovery                ] [SEA-2670-42.paraport.com] elasticsearch/NFlcTG-BThe2ADmZzLYHQw
[2015-12-23 11:47:39,266][INFO ][cluster.service          ] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][NFlcTG-BThe2ADmZzLYHQw][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-23 11:47:39,288][INFO ][http                     ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]}
[2015-12-23 11:47:39,289][INFO ][node                     ] [SEA-2670-42.paraport.com] started
[2015-12-23 11:47:39,353][INFO ][gateway                  ] [SEA-2670-42.paraport.com] recovered [1] indices into cluster_state

Here's my config:

################################### Cluster ###################################

cluster.name: elasticsearch

#################################### Node #####################################

node.name: SEA-2670-42.paraport.com
node.max_local_storage_nodes: 1

#################################### Index ####################################

index.mapper.dynamic: true
action.auto_create_index: true
action.disable_delete_all_indices: true

#################################### Paths ####################################

path.conf: /usr/local/etc/elasticsearch
path.data: /usr/local/var/data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch

#################################### Plugin ###################################


################################### Memory ####################################

bootstrap.mlockall: true

############################## Network And HTTP ###############################

network.host: SEA-2670-42.paraport.com
http.port: 9200

################################### Gateway ###################################

gateway.expected_nodes: 1

############################# Recovery Throttling #############################


################################## Discovery ##################################


discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.multicast.enabled: false

cloud.node.auto_attributes: true

Hope this helps.

11

[2015-12-23 11:47:39,266][INFO ][cluster.service ] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][NFlcTG-BThe2ADmZzLYHQw][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-23 11:47:39,288][INFO ][http ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]}

This show you are binding to localhost (127.0.0.1) on ports 9200 and 9300.

network.host: SEA-2670-42.paraport.com

Can you login to the shell on that host? On that host, if you "ping SEA-2670-42.paraport.com" what IP address do you get back?

I suspect your DNS or /etc/hosts is mapping the localhost address to that name.

12

Hey tinle,

An ifconfig on that box returns the same IP you get pinging the hostname from another machine. An nslookup on that IP returns the expected hostname. pinging itself in an SSH session returns the loopback IP, but thats to be expected. I'm not seeing any DNS wonkyness going on here.

13

I suspect you are running into the JVM DNS caching issue. Such as sometime in the past, that hostname got mapped to localhost and now it's stuck to always resolving to that for the JVM.

JVM DNS caching

14

So I rebuilt the entire machine since there was more manual touch at this point than I like. I've confirmed the config is identical to the one I posted above and this was the original config so localhost should not be cached. I can now curl the hostname:9200 (not localhost:9200) from itself but not from an external machine. I'm not running IP tables and I've gone ahead and disabled ufw (I'm on Ubuntu) and I'm still getting connection refused. Here's the log when I restart the service if it helps.

[2015-12-24 10:10:45,086][INFO ][node                     ] [SEA-2670-42.paraport.com] stopping ...
[2015-12-24 10:10:45,109][INFO ][node                     ] [SEA-2670-42.paraport.com] stopped
[2015-12-24 10:10:45,109][INFO ][node                     ] [SEA-2670-42.paraport.com] closing ...
[2015-12-24 10:10:45,116][INFO ][node                     ] [SEA-2670-42.paraport.com] closed
[2015-12-24 10:11:03,697][INFO ][node                     ] [SEA-2670-42.paraport.com] version[1.7.1], pid[22867], build[b88f43f/2015-07-29T09:54:16Z]
[2015-12-24 10:11:03,697][INFO ][node                     ] [SEA-2670-42.paraport.com] initializing ...
[2015-12-24 10:11:03,807][INFO ][plugins                  ] [SEA-2670-42.paraport.com] loaded [], sites []
[2015-12-24 10:11:03,855][INFO ][env                      ] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/template12-root)]], net usable_space [1.6gb], net total_space [3.9gb], types [ext4]
[2015-12-24 10:11:06,761][INFO ][node                     ] [SEA-2670-42.paraport.com] initialized
[2015-12-24 10:11:06,761][INFO ][node                     ] [SEA-2670-42.paraport.com] starting ...
[2015-12-24 10:11:06,838][INFO ][transport                ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9300]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9300]}
[2015-12-24 10:11:06,851][INFO ][discovery                ] [SEA-2670-42.paraport.com] elasticsearch/kLdZmqvfSrmqE45wKhM9QA
[2015-12-24 10:11:09,874][INFO ][cluster.service          ] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][kLdZmqvfSrmqE45wKhM9QA][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-24 10:11:09,899][INFO ][http                     ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]}
[2015-12-24 10:11:09,899][INFO ][node                     ] [SEA-2670-42.paraport.com] started
[2015-12-24 10:11:09,915][INFO ][gateway                  ] [SEA-2670-42.paraport.com] recovered [0] indices into cluster_state

Does it expect credentials or something? I've almost exhausted things to try at this point.

15

I even tried adding a line for http.host in the config since I am running 1.7.1 but that doesn't seem to have any effect.

16

The log still show the host is still resolving to 127.0.0.1

[2015-12-24 10:11:09,874][INFO ][cluster.service          ] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][kLdZmqvfSrmqE45wKhM9QA][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2015-12-24 10:11:09,899][INFO ][http                     ] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]}

Is that a physical host or VM (docker?). If you know its IP address, how about using that in the config instead of its name?

Alternatively, use '0.0.0.0'

E.g.

network.host: 0.0.0.0

17

This is a VM in VMWare vSphere.

I tried the IP before I switched to using the hostname and was getting the same results. Is there something wrong with my config? Why is it still using localhost? And if thats the case why am I able to curl it by hostname from itself but not localhost anymore?

root@SEA-2670-42:/usr/local/var/log/elasticsearch# curl SEA-2670-42.paraport.com:9200
{
  "status" : 200,
  "name" : "SEA-2670-42.paraport.com",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
root@SEA-2670-42:/usr/local/var/log/elasticsearch# curl http://localhost:9200
curl: (7) couldn't connect to host
18

I don't use vSphere so won't be much help here. My guess is that its resolver stack is doing some funny remapping internally, such that when you are in the VM, its hostname will resolve correctly, but using direct localhost address won't.

Best to ask this question in a VMWARE forum.

How about using 0.0.0.0 IP address? Does that not work? If it does not, I am out of ideas :frowning:

Sorry.

19

Hmm, I doubt this has anything to do with vSphere. Our application stack has all sorts of load balancing and endpoints all over the place and I have never had issues like this. I'll try 0.0.0.0 but how is that going to make it externally available?

20

And I apologize if that came across a bit rude. That was not my intention at all and I really appreciate the help. Just a bit frustrated that I'm still stuck on this :persevere: