No communication between Filebeat and Logstash

Elastic Stack Beats
1

Hi all !

I tried to install the ELK solution on a server and filebeat on my client, but it seems that there is no communication between the two...

I tried

beatname -c config.yml -e -d "*"
to help you but I get "command not found".

My config :

ELK Server

Client Server (the one I want to log)

Then I tested it with

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
on my ELK server but i get 0 hits...

Do you have any idea to find the issue ?

Thanks a lot !

Gisamark

2

which beat are you testing? filebeat? Then try $ filebeat -c <path-to-config-file> -e -d '*' in order to print some debug to console.

1 Like
3

I really recommend following the Filebeat getting started guide . And if you are going to use Logstash, see this short guide on how to configure Logstash for Beats.

If you are still having a problems after following the our guides, please post your versions (Filebeat/Logstash/Elasticsearch), configs, and log messages.

"beatname" <-- That's supposed to be "filebeat". Did you find that in documentation somewhere?

1 Like
4

Thank you.

I find that command on Elastic forum, "How to report an issue" :

The result of that command

filebeat -c config.yml -e -d "*"

gives me a LOT of text which I cannot copy here...

5

Ok,
I can see in the debug mode of filebeat :

2016/07/28 14:44:12.485085 publish.go:109: DBG  Publish: {
  "@timestamp": "2016-07-28T14:44:11.990Z",
  "beat": {
    "hostname": "localhost.localdomain",
    "name": "localhost.localdomain"
  },
  "count": 1,
  "fields": null,
  "input_type": "log",
  "message": "Jul 25 09:20:32 localhost /usr/bin/filebeat[2658]:
transport.go:125: SSL client failed to connect with: dial tcp 192.168.2.80:5044: getsockopt: connection refused",
  "offset": 310642,
  "source": "/var/log/messages",
  "type": "syslog"
}

So I guess i made a mistake with the SSL configuration ?

6

can you ping and telnet the logstash host? connection refused is not cause of SSL, but network.

7

I can ping and Telnet it.

But i copied the wrong message. The issue seems to be "EOF" :

2016/07/28 16:06:17.372218 output.go:87: DBG  output worker: publish 1024 events
2016/07/28 16:06:17.372303 client.go:146: DBG  Try to publish 1024 events to logstash with window size 10
2016/07/28 16:06:17.384023 client.go:105: DBG  close connection
2016/07/28 16:06:17.384189 client.go:124: DBG  0 events out of 1024 events sent to logstash. Continue sending ...
2016/07/28 16:06:17.384218 single.go:76: INFO Error publishing events (retrying): EOF
2016/07/28 16:06:17.384240 single.go:152: INFO send fail
2016/07/28 16:06:17.384259 single.go:159: INFO backoff retry: 1s
^C2016/07/28 16:06:17.666590 service.go:30: DBG  Received sigterm/sigint, stopping

Any idea ?

8

can you share beats output configuration and logstash input config?

1 Like
9

Filebeat server
/etc/filebeat/filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/secure"
        - "/var/log/messages"
        - "/var/log/*.log"

      input_type: log

      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:

  logstash:
    # The Logstash hosts
    hosts: ["192.168.2.80:5044"]
    bulk_max_size: 1024

    # Optional TLS. By default is off.
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

############################# Shipper #########################################
shipper:

############################# Logging #########################################
logging:

  # To enable logging to files, to_files option has to be set to true
  files:
    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

ELK Server
/etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
10

can you try without TLS first?

Check your certificate still being valid. I remember EOF being returned by the networking libs, if certificate has been expired.

1 Like
11

I tried without TLS and re-installed ELK according to elastic guide.
Now it works !

I'm not sure what happened, i will go without SSL for now.

I suggest users to follow the Elastic guides.

Thank you steffens and andrewkroh for your help ! :slight_smile:

12

This topic was automatically closed after 21 days. New replies are no longer allowed.