# Breaking up ELK stack to individual machines **URL:** \ **Category:** Elasticsearch\ **Created:** [December 22, 2015, 9:09pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796 "2015-12-22T21:09:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 22, 2015, 9:09pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/1 "2015-12-22T21:09:12Z")
Hello there, I am trying to break apart my current install of ELK that lives on one machine to an individual machine for Logstash, ElasticSearch, and Kibana. It seems as if connections to ES are being blocked as neither my Logstash or Kibana machine can connect to ES. I'm using IP and the default port (9200). I've set "network.host" to the IP of the ES node in the elasticsearch.yml file. If I understand correctly this is how you configure ES to be accessible on the address set in "network.host" to external machines, yet this doesn't seem to work. I'm not running any firewalls and if I put Kibana or Logstash on the same machine with ES installled it works fine. Any help would be greatly appreciated, thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 22, 2015, 9:17pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/2 "2015-12-22T21:17:48Z")
Can you curl to the ES host from a remote machine to see if it's accessible? ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 22, 2015, 9:18pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/3 "2015-12-22T21:18:49Z")
I get connection refused. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 22, 2015, 11:40pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/4 "2015-12-22T23:40:04Z")
Can you post the first few log lines from when ES starts up? ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 23, 2015, 12:03am UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/5 "2015-12-23T00:03:15Z")
``` [2015-12-22 09:19:46,896][INFO][node] [SEA-2670-42.paraport.com] stopping ... [2015-12-22 09:19:46,971][INFO][node] [SEA-2670-42.paraport.com] stopped [2015-12-22 09:19:46,971][INFO][node] [SEA-2670-42.paraport.com] closing ... [2015-12-22 09:19:46,988][INFO][node] [SEA-2670-42.paraport.com] closed [2015-12-22 09:20:38,945][INFO][node] [SEA-2670-42.paraport.com] version[1.7.1], pid[6014], build[b88f43f/2015-07-29T09:54:16Z] [2015-12-22 09:20:38,945][INFO][node] [SEA-2670-42.paraport.com] initializing ... [2015-12-22 09:20:39,056][INFO][plugins] [SEA-2670-42.paraport.com] loaded [], sites [] [2015-12-22 09:20:39,115][INFO][env] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/SEA--2670--87--vg-root)]], net usable_space [33.9gb], net total_space [38gb], types [ext4] [2015-12-22 09:20:42,310][WARN][script] [SEA-2670-42.paraport.com] deprecated setting [script.disable_dynamic] is set, replace with fine-grained scripting settings (e.g. script.inline, script.indexed, script.file) [2015-12-22 09:20:42,705][INFO][node] [SEA-2670-42.paraport.com] initialized [2015-12-22 09:20:42,706][INFO][node] [SEA-2670-42.paraport.com] starting ... [2015-12-22 09:20:42,790][INFO][transport] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[localhost/127.0.0.1:9300]} [2015-12-22 09:20:42,803][INFO][discovery] [SEA-2670-42.paraport.com] elasticsearch/Isb20mFoRDWWGtGSxlyMIQ [2015-12-22 09:20:45,829][INFO][cluster.service] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][Isb20mFoRDWWGtGSxlyMIQ][SEA-2670-42.paraport.com][inet[localhost/127.0.0.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master) [2015-12-22 09:20:45,852][INFO][http] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]} [2015-12-22 09:20:45,853][INFO][node] [SEA-2670-42.paraport.com] started [2015-12-22 09:20:45,883][INFO][gateway] [SEA-2670-42.paraport.com] recovered [1] indices into cluster_state ``` I looked at these logs before posting and found it odd that it didn't seem to reference the network.host config I changed, instead it just looks like its publishing on localhost. That being said, I don't really know how to interpret these logs. Thank you for your help! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 23, 2015, 12:04am UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/6 "2015-12-23T00:04:55Z")
> [@jmillerparaport](#): > > inet[/127.0.0.1:9200 It's still only listening on localhost, what is the config you have set? ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 23, 2015, 12:07am UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/7 "2015-12-23T00:07:49Z")
Everything in the config is commented out except for this line `network.host: 10.41.150.249` which is the IP of the machine. The documentation reads as if this is the only thing that needs to be set. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 23, 2015, 12:13am UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/8 "2015-12-23T00:13:03Z")
Try `network.host: 0.0.0.0` and restart it, you should see that inet part of the log change. ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 23, 2015, 12:19am UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/9 "2015-12-23T00:19:25Z")
I'm getting errors in the logs now after changing that. I'll need to take another look at this with fresh eyes in the morning. I might just rebuild the machine from scratch. Thanks again for your help! I'll report back here if I find the solution or need any additional assistance. ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 23, 2015, 7:55pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/10 "2015-12-23T19:55:35Z")
Alright so I'm still getting connection refused when trying to curl but the logs at least look like its publishing the hostname. I tried curling 9200 and 9300 ``` [2015-12-23 11:47:07,518][INFO][node] [SEA-2670-42.paraport.com] stopping ... [2015-12-23 11:47:07,562][INFO][node] [SEA-2670-42.paraport.com] stopped [2015-12-23 11:47:07,562][INFO][node] [SEA-2670-42.paraport.com] closing ... [2015-12-23 11:47:07,569][INFO][node] [SEA-2670-42.paraport.com] closed [2015-12-23 11:47:32,803][INFO][node] [SEA-2670-42.paraport.com] version[1.7.1], pid[9869], build[b88f43f/2015-07-29T09:54:16Z] [2015-12-23 11:47:32,803][INFO][node] [SEA-2670-42.paraport.com] initializing ... [2015-12-23 11:47:32,907][INFO][plugins] [SEA-2670-42.paraport.com] loaded [], sites [] [2015-12-23 11:47:32,959][INFO][env] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/SEA--2670--87--vg-root)]], net usable_space [33.5gb], net total_space [38gb], types [ext4] [2015-12-23 11:47:36,138][INFO][node] [SEA-2670-42.paraport.com] initialized [2015-12-23 11:47:36,139][INFO][node] [SEA-2670-42.paraport.com] starting ... [2015-12-23 11:47:36,224][INFO][transport] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9300]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9300]} [2015-12-23 11:47:36,238][INFO][discovery] [SEA-2670-42.paraport.com] elasticsearch/NFlcTG-BThe2ADmZzLYHQw [2015-12-23 11:47:39,266][INFO][cluster.service] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][NFlcTG-BThe2ADmZzLYHQw][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master) [2015-12-23 11:47:39,288][INFO][http] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]} [2015-12-23 11:47:39,289][INFO][node] [SEA-2670-42.paraport.com] started [2015-12-23 11:47:39,353][INFO][gateway] [SEA-2670-42.paraport.com] recovered [1] indices into cluster_state ``` Here's my config: ``` ################################### Cluster ################################### cluster.name: elasticsearch #################################### Node ##################################### node.name: SEA-2670-42.paraport.com node.max_local_storage_nodes: 1 #################################### Index #################################### index.mapper.dynamic: true action.auto_create_index: true action.disable_delete_all_indices: true #################################### Paths #################################### path.conf: /usr/local/etc/elasticsearch path.data: /usr/local/var/data/elasticsearch path.logs: /usr/local/var/log/elasticsearch #################################### Plugin ################################### ################################### Memory #################################### bootstrap.mlockall: true ############################## Network And HTTP ############################### network.host: SEA-2670-42.paraport.com http.port: 9200 ################################### Gateway ################################### gateway.expected_nodes: 1 ############################# Recovery Throttling ############################# ################################## Discovery ################################## discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.multicast.enabled: false cloud.node.auto_attributes: true ``` Hope this helps. ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [December 23, 2015, 8:14pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/11 "2015-12-23T20:14:29Z")
> [2015-12-23 11:47:39,266][INFO][cluster.service] [[SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com)] new\_master [[SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com)][NFlcTG-BThe2ADmZzLYHQw][[SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com)][inet[[SEA-2670-42.paraport.com/127.0.1.1:9300](http://SEA-2670-42.paraport.com/127.0.1.1:9300)]]{max\_local\_storage\_nodes=1}, reason: zen-disco-join (elected\_as\_master) > [2015-12-23 11:47:39,288][INFO][http] [[SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com)] bound\_address {inet[/127.0.1.1:9200]}, publish\_address {inet[[SEA-2670-42.paraport.com/127.0.1.1:9200](http://SEA-2670-42.paraport.com/127.0.1.1:9200)]} This show you are binding to localhost (127.0.0.1) on ports 9200 and 9300. > network.host: [SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com) Can you login to the shell on that host? On that host, if you "ping [SEA-2670-42.paraport.com](http://SEA-2670-42.paraport.com)" what IP address do you get back? I suspect your DNS or /etc/hosts is mapping the localhost address to that name. ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 23, 2015, 10:17pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/12 "2015-12-23T22:17:14Z")
Hey tinle, An ifconfig on that box returns the same IP you get pinging the hostname from another machine. An nslookup on that IP returns the expected hostname. pinging itself in an SSH session returns the loopback IP, but thats to be expected. I'm not seeing any DNS wonkyness going on here. ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [December 23, 2015, 10:34pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/13 "2015-12-23T22:34:42Z")
I suspect you are running into the JVM DNS caching issue. Such as sometime in the past, that hostname got mapped to localhost and now it's stuck to always resolving to that for the JVM. [JVM DNS caching](http://www.rgagnon.com/javadetails/java-0445.html) ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 24, 2015, 6:24pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/14 "2015-12-24T18:24:33Z")
So I rebuilt the entire machine since there was more manual touch at this point than I like. I've confirmed the config is identical to the one I posted above and this was the original config so localhost should not be cached. I can now curl the hostname:9200 (not localhost:9200) from itself but not from an external machine. I'm not running IP tables and I've gone ahead and disabled ufw (I'm on Ubuntu) and I'm still getting connection refused. Here's the log when I restart the service if it helps. ``` [2015-12-24 10:10:45,086][INFO][node] [SEA-2670-42.paraport.com] stopping ... [2015-12-24 10:10:45,109][INFO][node] [SEA-2670-42.paraport.com] stopped [2015-12-24 10:10:45,109][INFO][node] [SEA-2670-42.paraport.com] closing ... [2015-12-24 10:10:45,116][INFO][node] [SEA-2670-42.paraport.com] closed [2015-12-24 10:11:03,697][INFO][node] [SEA-2670-42.paraport.com] version[1.7.1], pid[22867], build[b88f43f/2015-07-29T09:54:16Z] [2015-12-24 10:11:03,697][INFO][node] [SEA-2670-42.paraport.com] initializing ... [2015-12-24 10:11:03,807][INFO][plugins] [SEA-2670-42.paraport.com] loaded [], sites [] [2015-12-24 10:11:03,855][INFO][env] [SEA-2670-42.paraport.com] using [1] data paths, mounts [[/ (/dev/mapper/template12-root)]], net usable_space [1.6gb], net total_space [3.9gb], types [ext4] [2015-12-24 10:11:06,761][INFO][node] [SEA-2670-42.paraport.com] initialized [2015-12-24 10:11:06,761][INFO][node] [SEA-2670-42.paraport.com] starting ... [2015-12-24 10:11:06,838][INFO][transport] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9300]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9300]} [2015-12-24 10:11:06,851][INFO][discovery] [SEA-2670-42.paraport.com] elasticsearch/kLdZmqvfSrmqE45wKhM9QA [2015-12-24 10:11:09,874][INFO][cluster.service] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][kLdZmqvfSrmqE45wKhM9QA][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master) [2015-12-24 10:11:09,899][INFO][http] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]} [2015-12-24 10:11:09,899][INFO][node] [SEA-2670-42.paraport.com] started [2015-12-24 10:11:09,915][INFO][gateway] [SEA-2670-42.paraport.com] recovered [0] indices into cluster_state ``` Does it expect credentials or something? I've almost exhausted things to try at this point. ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 24, 2015, 6:32pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/15 "2015-12-24T18:32:45Z")
I even tried adding a line for http.host in the config since I am running 1.7.1 but that doesn't seem to have any effect. ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [December 24, 2015, 6:48pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/16 "2015-12-24T18:48:00Z")
The log still show the host is still resolving to 127.0.0.1 ``` [2015-12-24 10:11:09,874][INFO][cluster.service] [SEA-2670-42.paraport.com] new_master [SEA-2670-42.paraport.com][kLdZmqvfSrmqE45wKhM9QA][SEA-2670-42.paraport.com][inet[SEA-2670-42.paraport.com/127.0.1.1:9300]]{max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master) [2015-12-24 10:11:09,899][INFO][http] [SEA-2670-42.paraport.com] bound_address {inet[/127.0.1.1:9200]}, publish_address {inet[SEA-2670-42.paraport.com/127.0.1.1:9200]} ``` Is that a physical host or VM (docker?). If you know its IP address, how about using that in the config instead of its name? Alternatively, use '0.0.0.0' E.g. `network.host: 0.0.0.0` ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 24, 2015, 6:56pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/17 "2015-12-24T18:56:27Z")
This is a VM in VMWare vSphere. I tried the IP before I switched to using the hostname and was getting the same results. Is there something wrong with my config? Why is it still using localhost? And if thats the case why am I able to curl it by hostname from itself but not localhost anymore? ``` root@SEA-2670-42:/usr/local/var/log/elasticsearch# curl SEA-2670-42.paraport.com:9200 { "status" : 200, "name" : "SEA-2670-42.paraport.com", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.1", "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build_timestamp" : "2015-07-29T09:54:16Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" } root@SEA-2670-42:/usr/local/var/log/elasticsearch# curl http://localhost:9200 curl: (7) couldn't connect to host ``` ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [December 24, 2015, 7:11pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/18 "2015-12-24T19:11:04Z")
I don't use vSphere so won't be much help here. My guess is that its resolver stack is doing some funny remapping internally, such that when you are in the VM, its hostname will resolve correctly, but using direct localhost address won't. Best to ask this question in a VMWARE forum. How about using 0.0.0.0 IP address? Does that not work? If it does not, I am out of ideas ☹ Sorry. ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 24, 2015, 7:13pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/19 "2015-12-24T19:13:25Z")
Hmm, I doubt this has anything to do with vSphere. Our application stack has all sorts of load balancing and endpoints all over the place and I have never had issues like this. I'll try 0.0.0.0 but how is that going to make it externally available? ---
**Author:** ![jmillerparaport](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmillerparaport/32/6791_2.png) [@jmillerparaport](https://discuss.elastic.co/u/jmillerparaport)\ **Post date:** [December 24, 2015, 8:07pm UTC](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796/20 "2015-12-24T20:07:47Z")
And I apologize if that came across a bit rude. That was not my intention at all and I really appreciate the help. Just a bit frustrated that I'm still stuck on this 😣 [Next page](https://discuss.elastic.co/t/breaking-up-elk-stack-to-individual-machines/37796.md?page=2)