Filebeat not publishing to logstash

Elastic Stack Beats
1

hi there i have filebeat alpha 5.0
when i try to publish it directly to elasticsearch everythin works fine
but when i change the output to logstash no data is comming through
and i get the following errors
2016-09-14T11:14:18+03:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=6471 libbeat.publisher.published_events=60 libbeat.logstash.publish.read_bytes=48 libbeat.logstash.published_and_acked_events=60 publish.events=60 registrar.state_updates=1110 libbeat.logstash.published_but_not_acked_events=16 libbeat.logstash.publish.read_errors=2 libbeat.logstash.call_count.PublishEvents=7
2016-09-14T11:14:43+03:00 ERR Failed to publish events caused by: EOF
2016-09-14T11:14:43+03:00 INFO Error publishing events (retrying): EOF
2016-09-14T11:14:48+03:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=56 libbeat.logstash.published_and_acked_events=56 registrar.state_updates=888 libbeat.logstash.publish.read_errors=1 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.read_bytes=30 publish.events=56 libbeat.logstash.publish.write_bytes=4747 libbeat.logstash.published_but_not_acked_events=10
2016-09-14T11:14:58+03:00 ERR Failed to publish events caused by: EOF
2016-09-14T11:14:58+03:00 INFO Error publishing events (retrying): EOF
2016-09-14T11:15:18+03:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.read_bytes=48 libbeat.logstash.call_count.PublishEvents=5 registrar.state_updates=888 libbeat.logstash.publish.write_bytes=15983 libbeat.logstash.published_and_acked_events=141 libbeat.publisher.published_events=141 libbeat.logstash.published_but_not_acked_events=18 publish.events=141
2016-09-14T11:15:23+03:00 ERR Failed to publish events caused by: EOF
2016-09-14T11:15:23+03:00 INFO Error publishing events (retrying): EOF
2016-09-14T11:15:35+03:00 ERR Failed to publish events caused by: EOF
2016-09-14T11:15:35+03:00 INFO Error publishing events (retrying): EOF
2016-09-14T11:15:48+03:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=7 libbeat.logstash.publish.write_bytes=5852 libbeat.publisher.published_events=57 libbeat.logstash.published_and_acked_events=57 libbeat.logstash.publish.read_errors=2 libbeat.logstash.published_but_not_acked_events=12 libbeat.logstash.publish.read_bytes=36 registrar.state_updates=1110 publish.events=57
2016-09-14T11:15:50+03:00 ERR Failed to publish events caused by: EOF
2016-09-14T11:15:50+03:00 INFO Error publishing events (retrying): EOF
2016-09-14T11:16:18+03:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=5389 libbeat.logstash.published_and_acked_events=58 libbeat.logstash.published_but_not_acked_events=22 registrar.state_updates=888 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.read_bytes=36 libbeat.logstash.call_count.PublishEvents=5 libbeat.publisher.published_events=58 publish.events=58
2016-09-14T11:16:48+03:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.published_and_acked_events=56 publish.events=56 libbeat.publisher.published_events=56 registrar.state_updates=1110 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.write_bytes=4172 libbeat.logstash.publish.read_bytes=30

2

this is my yml file

filebeat.prospectors:

- input_type: log

  
  paths:
    - /path/to/logs/1
  fields:
        log_type: "test1"
  ignore_older: "336h"
  multiline.pattern: '^(?:0?[1-9]|1[0-2])'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - /path/to/logs/2
  fields:
        log_type: "test2"
  ignore_older: "336h"
  multiline.pattern: '^(?:0?[1-9]|1[0-2])'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - /path/to/logs/3
  fields:
        log_type: "test3"
  ignore_older: "336h"
  multiline.pattern: '^(?:0?[1-9]|1[0-2])'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - /path/to/logs/4
  fields:
        log_type: "test4"
  ignore_older: "336h"
  multiline.pattern: '^(?:0?[1-9]|1[0-2])'
  multiline.negate: true
  multiline.match: after


output.logstash:
  hosts: ["logstashserver:5044"]

any help ?

3

please format your configs using the </> button. Beats config file format is sensitive to indentation and without proper formatting it's hard to read/check your config.

Seems the connection is closed by logstash. Check logstash logs. How is your logstash beats input configured?

4

I applied the formatting

5

Any update on this? I am seeing the same problem. In my case some of the data is making it to logstash, but not all of it.

6

@chris.mcdermott Can you please share some more details? LS / FB version, Config files, logs, ...?

7

This topic was automatically closed after 21 days. New replies are no longer allowed.