Logstash/Beats Issue - Logs not being published

[Danxx6 (Dan Lord)]

Hi

I've recently followed a couple of guides to get a ELK stack up and working however I'm having some issues transporting my logs via filebeat to logstash.

Errors:
2016-11-01T18:11:19Z ERR Failed to publish events caused by: EOF
2016-11-01T18:11:19Z INFO Error publishing events (retrying): EOF

Filebeat Config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
      #  - /var/log/*.log

      input_type: log

      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["elk:5044"]
    bulk_max_size: 1024

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

Logstash Config:

input {
  lumberjack {
    port => 5044
    type => "logs"
    ssl_certificate => "/etc/logstash/logstash-forwarder.crt"
    ssl_key => "/etc/logstash/logstash-forwarder.key"
  }
}

Does anyone know what's causing this?

Thanks

[andrewkroh]

What version of Filebeat are you using?

Did you follow the official Getting Started guide (5.X) provided by Elastic?

The Logstash input configuration you are using is not correct for use with Beats. See the Logstash Setup docs.

[Danxx6 (Dan Lord)]

Hi Andrew

Thanks for that, I've now figured it out and got it working. Just need to add the SSL now.

One question, in the tutorial i was following a separate file was created in the /etc/logstash/conf.d/ folder for the syslog filter, do I need to use a separate file or does this go into the logstash.conf file now?

Thanks

[andrewkroh]

@Danxx6

One question, in the tutorial i was following a separate file was created in the /etc/logstash/conf.d/ folder for the syslog filter, do I need to use a separate file or does this go into the logstash.conf file now?

All of the config files in the conf.d directory are concatenated in lexicographical order and then parsed as a single config file. So you can split them up or just have one large file.