How to make filebeat multiline plugin send the unmatched logs

Elastic Stack Beats
1

Hi, I am using filebeat 5.6.X version
With the below config, filebeat sends five lines to elasticsearch from below log file, using multi line pattern. But the first line " host down true wiley-host" is dropped. How to get the line "host down true wiley-host" into elasticsearch as a separate document. any suggestions??

Below is the actual log file i am parsing
host down true wiley-host
Alert high CPU status on host wiley-host
cannot reach the host. Port not reachable
host down form twenty minutes
host status unknown for last twenty minutes
service execution failed with exit status 22

Below is my filebeat config

filebeat:
  prospectors:
  - input_type: log
    paths:
    - "/var/lib/host/wiley.log"
    fields:
      type: hostlog
    fields_under_root: true
    multiline.pattern: '(exit status 22$)'
    multiline.negate: true
    multiline.match: before
    multiline.max_lines: 5
output:
  kafka:
    hosts:
    - host1.skg.com:29092
    - host2.skg.com:39092
    - host3.skg.com:49092
    topic: logs-%{[type]}

output in kibana

message: Alert high CPU status on host wiley-host
cannot reach the host. Port not reachable
host down form twenty minutes
host status unknown for last twenty minutes
service execution failed with exit status 22

Note: I am not using the filter pattern on logstash for the reason(below) mentioned in logstash docs

If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.

2

What do you mean by 'dropped'?

Why do you set max_lines: 5? The clips the multiline event after 5 lines. Lines afterwards, but still missing the pattern will not be send.

Do you have a more complate log file? With messages before/after?

3

@steffens

What do you mean by 'dropped'?
I want the first line of the log file "host down true wiley-host" also to be sent to elasticsearch and view the message in kibana as a new document.

Why do you set max_lines: 5?

I want to store the matching line that is "service execution failed with exit status 22" and the four lines above it as a single document.

Below is the log file with before and after

host down true wiley-host
Alert high CPU status on host wiley-host
cannot reach the host. Port not reachable
host down form twenty minutes
host status unknown for last twenty minutes
service execution failed with exit status 22
retrying to execute service "ping-check"....
"ping-check" failed with exit status 22

Are you saying that the lines missing the pattern will not be send to elasticsearch?

4

This is not how max_lines works. One configures one pattern and depending on negate setting, all lines (not) matching will be combined into one event. While matching multiple lines until flush, only the first max_lines will be published. The first host down true wiley-host should still be shipped, but not dropped.

I assume you don't have any other lines before or after your file. That is basically one file per status check. Plus I assume you want to generate 2 events:

host down true wiley-host
Alert high CPU status on host wiley-host
cannot reach the host. Port not reachable
host down form twenty minutes
host status unknown for last twenty minutes
service execution failed with exit status 22

and

retrying to execute service "ping-check"....
"ping-check" failed with exit status 22

Using these multiline settings:

  multiline.pattern: 'exit status \d+$'
  multiline.negate: true
  multiline.match: before

works well for me:

{
  "@timestamp": "2018-02-08T23:20:06.934Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "message": "host down true wiley-host\nAlert high CPU status on host wiley-host\ncannot reach the host. Port not reachable\nhost down form twenty minutes\nhost status unknown for last twenty minutes\nservice execution failed with exit status 22",
  "source": "/tmp/testinput.txt",
  "offset": 228,
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "xxx",
    "hostname": "xxx",
    "version": "7.0.0-alpha1"
  }
}
{
  "@timestamp": "2018-02-08T23:20:06.934Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "message": "retrying to execute service \"ping-check\"....\n\"ping-check\" failed with exit status 22",
  "input": {
    "type": "log"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "xxx",
    "hostname": "xxx",
    "version": "7.0.0-alpha1"
  },
  "source": "/tmp/testinput.txt",
  "offset": 313
}

Just do not use max_lines: 5.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.