Filebeat multiline directly to elasticsearch

[Matus_Gajdos]

Hello,

I would like to send application logs from filebeat directly to elasticsearch index.

I'm trying send multiline message but in index I see it as single line.

EmbargoServiceLogger Error: 92 : 3/12/2019 1:46:29 PM##4784##8100##Creation of listner #7 failed. Exception:System.IO.FileNotFoundException: Error reading the  Test\To\AllianceManual\Screening\SRC directory.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at EmbargoSvc.AMLEmbargoService.StartFileImport()

my filebeat.yml multiline configuration is following:

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

Could somebody help me with this?

Thx

[shaunak (Shaunak Kashyap)]

Hi @Matus_Gajdos,

Welcome!

I tried your settings and sample message with my Filebeat and I'm seeing a multiline (\n-delimited) message in Kibana Discover:

What are you seeing in your Kibana Discover? Alternatively, you could just post the complete Elasticsearch document for that log entry over here as well.

[Matus_Gajdos]

Hello @shaunak

here is what I see in kibana

Could you send me your filebeat configuration?
Are you sending data directly to elasticsearch or through logstash?

Thank you.

[shaunak (Shaunak Kashyap)]

So, first, I tested with the console output, just for debugging purposes. For that, my filebeat.yml looked like this:

filebeat.inputs:

- type: stdin
  enabled: true

  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after

output.console:
  enabled: true

Using that configuration, when I supplied your sample input on STDIN, I got the following output on STDOUT (in the console):

{"@timestamp":"2019-03-20T11:51:03.693Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.0.0"},"message":"EmbargoServiceLogger Error: 92 : 3/12/2019 1:46:29 PM##4784##8100##Creation of listner #7 failed. Exception:System.IO.FileNotFoundException: Error reading the  Test\\To\\AllianceManual\\Screening\\SRC directory.\n   at System.IO.FileSystemWatcher.StartRaisingEvents()\n   at EmbargoSvc.AMLEmbargoService.StartFileImport()","input":{"type":"stdin"},"ecs":{"version":"1.0.0"},"host":{"name":"Shaunaks-MBP-2"},"agent":{"version":"8.0.0","type":"filebeat","ephemeral_id":"1bc476cb-af43-440b-8626-61184b65792b","hostname":"Shaunaks-MBP-2","id":"51a8bb52-16b0-4129-a33b-9fba504fe6e8"},"log":{"flags":["multiline"],"offset":0,"file":{"path":""}}}

Note the \n characters in the message field.

Next, I changed the output to elasticsearch and provided the same input via STDIN. In that case I got what I posted earlier in the Kibana screenshot.

Are you sending via Logstash? Could you try the console output first like I did, just for debugging purposes?