Elasticsearch: Couldn't connect to any of the configured Elasticsearch hosts - connection refused]

Hello,
I have to virtualmachine with ubunu 19.10 in my local lab.

Host A: 192.168.29.90
/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  paths:
    - /opt/zeek/logs/current/*.log
filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
tags: ["filebeatzeek"]
setup.kibana:
  host: "192.168.29.87:5601"
output.elasticsearch:
   hosts: ["192.168.29.87:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

HostB: 192.168.29.87
/etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200

/etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"

Check with netstat de network

root@HostB:/home/# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      812/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      913/sshd            
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      865/node            
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 127.0.0.1:50440         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50446         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50438         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50436         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50444         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50442         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50448         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:50434         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0    196 192.168.29.87:22        192.168.29.10:57736     ESTABLISHED 2270/sshd: usuario  
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      2450/java           
tcp6       0      0 :::5044                 :::*                    LISTEN      842/java            
tcp6       0      0 :::22                   :::*                    LISTEN      913/sshd            
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      842/java            
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50442         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50436         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50448         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50444         ESTABLISHED 2450/java           
tcp6       1      0 127.0.0.1:46854         127.0.0.1:9200          CLOSE_WAIT  842/java            
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50446         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50434         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50440         ESTABLISHED 2450/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:50438         ESTABLISHED 2450/java 

Check the comunication front A > to B.

root@HostA:/home/# sudo filebeat setup
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.29.87:9200: Get http://192.168.29.87:9200: dial tcp 192.168.29.87:9200: connect: connection refused]

root@HostA:/home/# nc -vn 192.168.29.87 9200
nc: connect to 192.168.29.87 port 9200 (tcp) failed: Connection refused

root@HostA:/home/# nc -vn 192.168.29.87 5601
Connection to 192.168.29.87 5601 port [tcp/*] succeeded!

check on HostB witch curl
root@hostB:/home/usuario# curl -X GET "localhost:9200/?pretty"
{
"name" : "hostb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "RpVuwifoQEqgbEPAtu_0ow",
"version" : {
"number" : "7.5.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
"build_date" : "2019-11-26T01:06:52.518245Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
....................................

check on HostA witch curl

root@hostA:/home/# curl -X GET "192.168.29.87:9200/?pretty"
curl: (7) Failed to connect to 192.168.29.87 port 9200: Connection refused

Other information:

  • ELK version: 7.5
  • SO: ubuntu 19.10
  • Firewall: none

What I can change to resolve the wrong situation and send data to HostA to HostB using elasticsearch?

Why I can connect to 9200 on localhost but I can't from remote host?

How I can confire elasticsearch to listen from the port 9200 from all remote ips?

I am try to deploy Zeek (SIEM) from the "Getting Started" to use "Zeek logs". Hosta is Zeek, and HostB is ELK.

Thank

Why I can connect to 9200 on localhost but I can't from remote host?

This is because you have configured elasticsearch to listen to only localhost by setting the following value in elasticsearch.yml

network.host: localhost

How I can confire elasticsearch to listen from the port 9200 from all remote ips?

Change network.host value in elasticsearch.yml as below:

network.host: 0.0.0.0

This will make elasticsearch to listen on all available network interface.

Hi @Opster_Community1

When I change that and re-start elasticsearch service it failed.
After:

network.host: localhost

Before:

network.host: 0.0.0.0

Restart the service:

root@HostB:/home/# systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Checking journalctl -xe

Checking systemctl status elasticsearch.service

root@HostB:/home/# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2019-12-14 10:07:18 CET; 38s ago
     Docs: http://www.elastic.co
  Process: 3760 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 3760 (code=exited, status=78)

dic 14 10:06:58 HostB systemd[1]: Starting Elasticsearch...
dic 14 10:06:59 HostB elasticsearch[3760]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
dic 14 10:07:18 HostB elasticsearch[3760]: ERROR: [1] bootstrap checks failed
dic 14 10:07:18 HostB elasticsearch[3760]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
dic 14 10:07:18 HostB systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
dic 14 10:07:18 HostB systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
dic 14 10:07:18 HostB systemd[1]: Failed to start Elasticsearch.

I don't understand where is the error, but I uncommend two lines (in ELK 7.4 it will not necessary) and I reststart the service. RESOLVED. Elasticsearch up :grinning:

Why this lines are now necesaries?

discovery.seed_hosts: ["host1"]
cluster.initial_master_nodes: ["node-1"]

If I check netststat

root@HostB:/home/# netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      812/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      913/sshd            
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      865/node            
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      996/nginx: master p 
tcp        0      0 127.0.0.1:57614         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57618         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57608         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57606         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57620         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57610         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57612         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0      0 127.0.0.1:57616         127.0.0.1:9200          ESTABLISHED 865/node            
tcp        0    196 192.168.29.87:22        192.168.29.10:57736     ESTABLISHED 2270/sshd: usuario  
tcp6       0      0 :::9300                 :::*                    LISTEN      3928/java           
tcp6       0      0 :::5044                 :::*                    LISTEN      842/java            
tcp6       0      0 :::22                   :::*                    LISTEN      913/sshd            
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      842/java            
tcp6       0      0 :::9200                 :::*                    LISTEN      3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57618         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57614         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57610         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57606         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57608         ESTABLISHED 3928/java           
tcp6       1      0 127.0.0.1:46854         127.0.0.1:9200          CLOSE_WAIT  842/java            
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57616         ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57612         ESTABLISHED 3928/java           
tcp6       0      0 192.168.29.87:9200      192.168.29.90:43368     ESTABLISHED 3928/java           
tcp6       0      0 127.0.0.1:9200          127.0.0.1:57620         ESTABLISHED 3928/java      

I check withn lsof -i -P -n | grep elasticsearch

root@HostB:/home/# lsof -i -P -n | grep elasticsearch
java      3928   elasticsearch  206u  IPv6  76688      0t0  TCP *:9300 (LISTEN)
java      3928   elasticsearch  219u  IPv6  75442      0t0  TCP *:9200 (LISTEN)
java      3928   elasticsearch  238u  IPv6  76706      0t0  TCP 127.0.0.1:9200->127.0.0.1:57606 (ESTABLISHED)
java      3928   elasticsearch  241u  IPv6  76714      0t0  TCP 127.0.0.1:9200->127.0.0.1:57612 (ESTABLISHED)
java      3928   elasticsearch  244u  IPv6  76709      0t0  TCP 127.0.0.1:9200->127.0.0.1:57608 (ESTABLISHED)
java      3928   elasticsearch  245u  IPv6  76712      0t0  TCP 127.0.0.1:9200->127.0.0.1:57610 (ESTABLISHED)
java      3928   elasticsearch  247u  IPv6  76717      0t0  TCP 127.0.0.1:9200->127.0.0.1:57614 (ESTABLISHED)
java      3928   elasticsearch  251u  IPv6  76720      0t0  TCP 127.0.0.1:9200->127.0.0.1:57616 (ESTABLISHED)
java      3928   elasticsearch  252u  IPv6  76723      0t0  TCP 127.0.0.1:9200->127.0.0.1:57618 (ESTABLISHED)
java      3928   elasticsearch  253u  IPv6  76726      0t0  TCP 127.0.0.1:9200->127.0.0.1:57620 (ESTABLISHED)
java      3928   elasticsearch  254u  IPv6  76906      0t0  TCP 192.168.29.87:9200->192.168.29.90:43368 (ESTABLISHED)

Thanks.

The reason for this is that elasticsearch need some information to form the cluster.

discovery.seed_hosts tells which all nodes are master eligible and cluster.initial_master_nodes tells that votes of which all nodes will be considered toin election for the very first time.

To understand more about it in details read this article.

Also

You mentioned this as Before. Didn't this work?

Yes. I resolved it.
But you need configure that:

network.host: 0.0.0.0
discovery.seed_hosts: ["host1"]
cluster.initial_master_nodes: ["node-1"]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.