Elasticsearch: Couldn't connect to any of the configured Elasticsearch hosts - connection refused]

carmezsa · December 14, 2019, 3:02am

Hello,
I have to virtualmachine with ubunu 19.10 in my local lab.

Host A: 192.168.29.90
/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  paths:
    - /opt/zeek/logs/current/*.log
filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
tags: ["filebeatzeek"]
setup.kibana:
  host: "192.168.29.87:5601"
output.elasticsearch:
   hosts: ["192.168.29.87:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

HostB: 192.168.29.87
/etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200

/etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"

Check with netstat de network

root@HostB:/home/# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      812/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      913/sshd            
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      865/node            
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 127.0.0.1:50440         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50446         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50438         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50436         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50444         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50442         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50448         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50434         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0    196 192.168.29.87:22        192.168.29.10:57736     ESTABLISHED 2270/sshd: usuario  
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      2450/java           
tcp6       0      0 :::5044                 :::*                    LISTEN      842/java            
tcp6       0      0 :::22                   :::*                    LISTEN      913/sshd            
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      842/java            
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50442         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50436         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50448         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50444         ESTABLISHED 2450/java           
tcp6       1      0 127.0.0.1:46854         127.0.0.1:9200          CLOSE_WAIT  842/java            
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50446         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50434         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50440         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50438         ESTABLISHED 2450/java

Check the comunication front A > to B.

root@HostA:/home/# sudo filebeat setup
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.29.87:9200: Get http://192.168.29.87:9200: dial tcp 192.168.29.87:9200: connect: connection refused]

root@HostA:/home/# nc -vn 192.168.29.87 9200
nc: connect to 192.168.29.87 port 9200 (tcp) failed: Connection refused

root@HostA:/home/# nc -vn 192.168.29.87 5601
Connection to 192.168.29.87 5601 port [tcp/*] succeeded!

check on HostB witch curl
root@hostB:/home/usuario# curl -X GET "localhost:9200/?pretty"
{
"name" : "hostb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "RpVuwifoQEqgbEPAtu_0ow",
"version" : {
"number" : "7.5.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
"build_date" : "2019-11-26T01:06:52.518245Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
....................................

check on HostA witch curl

root@hostA:/home/# curl -X GET "192.168.29.87:9200/?pretty"
curl: (7) Failed to connect to 192.168.29.87 port 9200: Connection refused

Other information:

ELK version: 7.5
SO: ubuntu 19.10
Firewall: none

What I can change to resolve the wrong situation and send data to HostA to HostB using elasticsearch?

Why I can connect to 9200 on localhost but I can't from remote host?

How I can confire elasticsearch to listen from the port 9200 from all remote ips?

I am try to deploy Zeek (SIEM) from the "Getting Started" to use "Zeek logs". Hosta is Zeek, and HostB is ELK.

Thank

Opster_Community1 · December 14, 2019, 7:17am

Why I can connect to 9200 on localhost but I can't from remote host?

This is because you have configured elasticsearch to listen to only localhost by setting the following value in elasticsearch.yml

network.host: localhost

How I can confire elasticsearch to listen from the port 9200 from all remote ips?

Change network.host value in elasticsearch.yml as below:

network.host: 0.0.0.0

This will make elasticsearch to listen on all available network interface.

carmezsa · December 14, 2019, 9:25am

Hi @Opster_Community1

When I change that and re-start elasticsearch service it failed.
After:

network.host: localhost

Before:

network.host: 0.0.0.0

Restart the service:

root@HostB:/home/# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Checking journalctl -xe

Checking systemctl status elasticsearch.service

root@HostB:/home/# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2019-12-14 10:07:18 CET; 38s ago
     Docs: http://www.elastic.co
  Process: 3760 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 3760 (code=exited, status=78)

dic 14 10:06:58 HostB systemd[1]: Starting Elasticsearch...
dic 14 10:06:59 HostB elasticsearch[3760]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
dic 14 10:07:18 HostB elasticsearch[3760]: ERROR: [1] bootstrap checks failed
dic 14 10:07:18 HostB elasticsearch[3760]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
dic 14 10:07:18 HostB systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
dic 14 10:07:18 HostB systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
dic 14 10:07:18 HostB systemd[1]: Failed to start Elasticsearch.

I don't understand where is the error, but I uncommend two lines (in ELK 7.4 it will not necessary) and I reststart the service. RESOLVED. Elasticsearch up

Why this lines are now necesaries?

discovery.seed_hosts: ["host1"]
cluster.initial_master_nodes: ["node-1"]

If I check netststat

root@HostB:/home/# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      812/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      913/sshd            
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      865/node            
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 127.0.0.1:57614         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57618         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57608         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57606         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57620         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57610         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57612         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57616         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0    196 192.168.29.87:22        192.168.29.10:57736     ESTABLISHED 2270/sshd: usuario  
tcp6       0      0 :::9300                 :::*                    LISTEN      3928/java           
tcp6       0      0 :::5044                 :::*                    LISTEN      842/java            
tcp6       0      0 :::22                   :::*                    LISTEN      913/sshd            
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      842/java            
tcp6       0      0 :::9200                 :::*                    LISTEN      3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57618         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57614         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57610         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57606         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57608         ESTABLISHED 3928/java           
tcp6       1      0 127.0.0.1:46854         127.0.0.1:9200          CLOSE_WAIT  842/java            
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57616         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57612         ESTABLISHED 3928/java           
tcp6       0      0 192.168.29.87:9200      192.168.29.90:43368     ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57620         ESTABLISHED 3928/java

I check withn lsof -i -P -n | grep elasticsearch

root@HostB:/home/# lsof -i -P -n | grep elasticsearch
java      3928   elasticsearch  206u  IPv6  76688      0t0  TCP *:9300 (LISTEN)
java      3928   elasticsearch  219u  IPv6  75442      0t0  TCP *:9200 (LISTEN)
java      3928   elasticsearch  238u  IPv6  76706      0t0  TCP 127.0.0.1:9200->127.0.0.1:57606 (ESTABLISHED)
java      3928   elasticsearch  241u  IPv6  76714      0t0  TCP 127.0.0.1:9200->127.0.0.1:57612 (ESTABLISHED)
java      3928   elasticsearch  244u  IPv6  76709      0t0  TCP 127.0.0.1:9200->127.0.0.1:57608 (ESTABLISHED)
java      3928   elasticsearch  245u  IPv6  76712      0t0  TCP 127.0.0.1:9200->127.0.0.1:57610 (ESTABLISHED)
java      3928   elasticsearch  247u  IPv6  76717      0t0  TCP 127.0.0.1:9200->127.0.0.1:57614 (ESTABLISHED)
java      3928   elasticsearch  251u  IPv6  76720      0t0  TCP 127.0.0.1:9200->127.0.0.1:57616 (ESTABLISHED)
java      3928   elasticsearch  252u  IPv6  76723      0t0  TCP 127.0.0.1:9200->127.0.0.1:57618 (ESTABLISHED)
java      3928   elasticsearch  253u  IPv6  76726      0t0  TCP 127.0.0.1:9200->127.0.0.1:57620 (ESTABLISHED)
java      3928   elasticsearch  254u  IPv6  76906      0t0  TCP 192.168.29.87:9200->192.168.29.90:43368 (ESTABLISHED)

Thanks.

Opster_Community1 · December 14, 2019, 10:07am

The reason for this is that elasticsearch need some information to form the cluster.

discovery.seed_hosts tells which all nodes are master eligible and cluster.initial_master_nodes tells that votes of which all nodes will be considered toin election for the very first time.

To understand more about it in details read this article.

Opster_Community1 · December 14, 2019, 10:13am

Also

You mentioned this as Before. Didn't this work?

carmezsa · December 14, 2019, 6:33pm

Yes. I resolved it.
But you need configure that:

network.host: 0.0.0.0
discovery.seed_hosts: ["host1"]
cluster.initial_master_nodes: ["node-1"]

system · January 11, 2020, 6:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.