Filebeat not sending empty lines

Elastic Stack Beats
1

Hi all,

I looked around and it seems like filebeat should be sending empty lines in the log files. Running version 6.1.3 with a pretty basic configuration but it looks like the harvester is ignoring them. Am I missing something?

Config file:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /usr/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Test file:

hello


the lines above are ignored.

Run with command line (debug mode):

/usr/share/filebeat/bin/filebeat -e -d "*" -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Events published (notice empty lines not published):

2018/02/01 15:55:11.560440 processor.go:275: DBG [publish] Publish event: {
  "@timestamp": "2018-02-01T15:55:11.560Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.1.3"
  },
  "offset": 6,
  "message": "hello",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "myhost",
    "hostname": "myhost",
    "version": "6.1.3"
  },
  "source": "/usr/test2.log"
}
2018/02/01 15:55:11.560507 processor.go:275: DBG [publish] Publish event: {
  "@timestamp": "2018-02-01T15:55:11.560Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.1.3"
  },
  "source": "/usr/test2.log",
  "offset": 37,
  "message": "the lines above are ignored.",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "hostname": "myhost",
    "version": "6.1.3",
    "name": "myhost"
  }
}

Thank you.

2

Are you suggesting that those are part of the same event, and that you have multiline messages with empty CR/LFs in them?

3

For the above I simply copied the test file into the directory being watched by FileBeat to produce those results. From what I've seen I generally have events that consist of single empty lines (e.g. a program may simply 'printf("\n")' into a log file). I am trying to reconstruct that log file at the receiving end, but FileBeat is leaving out the newlines so my files aren't matching.

4

Should I open an issue in github before this thread auto closes?

5

It seems to be indeed the case that empty lines are ignored. But what is your use case for needing empty lines? Seems that in the logging use case they are not exactly useful.

Playing around with the multiline config, it is possible to append the empty lines to the previous line, so that they don't create a a dedicated event, but they are not completely lost either. This is the multiline config that I tried:

  multiline.pattern: '^$'
  multiline.match: after
6

Thank you for looking. I will try the multiline pattern. Our use case is to rebuild the log files to their original format after processing. The empty lines keep the formatting of the file intact and make it easier to read should we need to examine them later.

The documentation seems to indicate these lines shouldn't be ignored. Is there something to be fixed or should the documentation be updated?

7

Can you open a ticket to report the documentation issue, please?

8

This topic was automatically closed after 21 days. New replies are no longer allowed.

10

https://github.com/elastic/beats/issues/6476 Thank you.

1 Like