Error connecting to Elasticsearch messages while starting filebeat

Elastic Stack Beats
1

Dear All,
my instance details are

Kibana version: 7.4.1
Elasticsearch version: 7.4.1
**filebeat version ** 7.4.1
APM Agent language and version: NA
Logstash version 7.4.1-1

From my clients, I can telnet to the port 5044 of the elasticsearch server

telnet 192.168.3.191 5044
Trying 192.168.3.191...
Connected to 192.168.3.191.
Escape character is '^]'.

ie logstash port is open, now from the client machine when I run filebeat I am getting the following error

|019-10-30T11:51:36.757+0400|ERROR|elasticsearch/elasticsearch.go:260|Error connecting to Elasticsearch at http://192.168.3.191:5044: Get http://192.168.3.191:5044: read tcp 10.212.135.200:55782->192.168.3.191:5044: read: connection reset by peer|

|2019-10-30T11:51:36.757+0400|ERROR|fileset/factory.go:131|Error loading pipeline: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.3.191:5044: Get http://192.168.3.191:5044: read tcp 10.212.135.200:55782->192.168.3.191:5044: read: connection reset by peer]|

|2019-10-30T11:51:36.786+0400|ERROR|elasticsearch/elasticsearch.go:260|Error connecting to Elasticsearch at http://192.168.3.191:5044: Get http://192.168.3.191:5044: read tcp 10.212.135.200:55784->192.168.3.191:5044: read: connection reset by peer|

When I observer this message, I could see an " 10.212.135.200", why this IP is refered, I am doing the setup in my LAN enviornment, and I do not have a public IP address and how this 10.212.135.200 us coming in my log files

Advise requested to know why this IP occurrence and why my filebeat is not able to communicate to the logstash server (logstash server uou can telnet to port 5044)

thanks
Joseph John

2

Like to update
when I give route -n

route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.10.1 0.0.0.0 UG 600 0 0 wlp3s0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 virbr0
192.168.3.0 10.212.135.200 255.255.255.0 UG 0 0 0 ppp0
192.168.10.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp3s0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
217.165.140.179 192.168.10.1 255.255.255.255 UGH 0 0 0 wlp3s0

So it is reaching out to the gateway throuhh pp0

3

Based on the errors you have shared it seems that Filebeat is trying to Elasticsearch not Logstash. I assume you accidentally configured the ES output. But it is difficult to say anything without a configuration. Could you please share your configuration and format it using </>?

1 Like
4

Hi Noemi,
thanks, I am posting the conf files, first section I am posting the server conf and then at the clients side

Server Conf Details

Elasticsearch
cat /etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.type: single-node
network.host: 192.168.3.191

Kibana
cat /etc/kibana/kibana.yml

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.3.191:9200"]

LogStash (at elasticsearch server)
cat /etc/logstash/logstash.yml

path.data: /var/lib/logstash
http.host: "192.168.3.191"
http.port: 9600-9700
log.level: error
path.logs: /var/log/logstash

from the client for the filebeat and auditbeat
/etc/auditbeat/auditbeat.yml

auditbeat.modules:

  • module: auditd
    audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
    audit_rules: |
  • module: file_integrity
    paths:
    • /bin
    • /usr/bin
    • /sbin
    • /usr/sbin
    • /etc
  • module: system
    datasets:
    • host # General host information, e.g. uptime, IPs
    • login # User logins, logouts, and system boots.
    • package # Installed, updated, and removed packages
    • process # Started and stopped processes
    • socket # Opened and closed sockets
    • user # User information
      state.period: 12h
      user.detect_password_changes: true
      login.wtmp_file_pattern: /var/log/wtmp*
      login.btmp_file_pattern: /var/log/btmp*
      setup.template.settings:
      index.number_of_shards: 1
      setup.kibana:
      host: "192.168.3.191:5601"
      > output.elasticsearch:
      > hosts: ["192.168.3.191:9200"]
      > hosts: ["192.168.3.191:5044"]
      processors:
    • add_host_metadata: ~
    • add_cloud_metadata: ~
      logging.level: error

from the client filebeat
/etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "192.168.3.191:5601"
output.elasticsearch:
  hosts: ["192.168.3.191:9200"]
  hosts: ["192.168.3.191:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
logging.level: error
5

thanks for the lead , now I commented out the output to elasticsearch and enabled output to logstash and it is working now

output.logstash:
hosts: ["192.168.3.191:5044"]

THANKS A LOT

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.