Cannot parse multiline log using Filebeat

[Bhanu1 (Bhanu)]

Hi

I am new to Filebeats , I need some help to parse logfiles with below pattern

Please help

2020-03-27T14:00:05+00:00 log-forwarder-68kh5 kapacitor-7cf7f7bdd4-lb6cn_monit {"log":"ts=2020-03-27T14:00:05.321Z lvl=info msg="http request" service=http host=127.0.0.1 username=- start=2020-03-27T14:00:05.319802032Z method=GET uri=/kapacitor/v1/task

s?dot-view=attributes\u0026fields=type\u0026fields=status\u0026fields=executing\u0026fields=dbrps\u0026limit=100\u0026offset=0\u0026pattern=\u0026replay-id=\u0026script-format=formatted protocol=HTTP/1.1 status=200 referer=- user-agent=KapacitorClient re

quest-id=42b7acc8-7033-11ea-855b-000000000000 duration=1.505141ms\n","stream":"stderr","time":"2020-03-27T14:00:05.321393951Z"}

2020-03-27T14:00:09+00:00 log-forwarder-68kh5 kapacitor-7cf7f7bdd4-lb6cn_monit {"log":"ts=2020-03-27T14:00:08.990Z lvl=info msg="http request" service=http host=10.244.52.203 username=- start=2020-03-27T14:00:08.990170521Z method=POST uri=/write?consis

tency=\u0026db=k8s\u0026precision=ns\u0026rp=default protocol=HTTP/1.1 status=204 referer=- user-agent=InfluxDBClient request-id=44e7ba8e-7033-11ea-8561-000000000000 duration=180.931µs\n","stream":"stderr","time":"2020-03-27T14:00:08.990597135Z"}

2020-03-27T14:00:09+00:00 log-forwarder-68kh5 kapacitor-7cf7f7bdd4-lb6cn_monit {"log":"ts=2020-03-27T14:00:09.022Z lvl=info msg="http request" service=http host=10.244.52.203 username=- start=2020-03-27T14:00:09.022182012Z method=POST uri=/write?consis

tency=\u0026db=k8s\u0026precision=ns\u0026rp=default protocol=HTTP/1.1 status=204 referer=- user-agent=InfluxDBClient request-id=44ec9d01-7033-11ea-8562-000000000000 duration=86.487µs\n","stream":"stderr","time":"2020-03-27T14:00:09.023165862Z"}

[ChrsMark (Chris Mark)]

Hi @Bhanu1!

First of all you should check if the service from which you want to collect and parse the logs is already supported by a Filebeat module. See modules list: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html

If the service cannot be covered by a Filebeat module then you should consider sending the logs with Filebeat to Logstash and analyze them there using your own patterns.

[Bhanu1 (Bhanu)]

Thanks @ChrsMark , Yes this is supported by Filebeat

[Bhanu1 (Bhanu)]

All i need is how to parse above messages

[ChrsMark (Chris Mark)]

@Bhanu1 please find the module that can support this service (btw what is it?) and follow its documentation so as to familiarise yourself with its usage and have it properly configured in order to work for your case.

Thank you!

[Bhanu1 (Bhanu)]

Hi
Sorry custom Module name is mule and is not working , Any way to parse strings like above format

[Bhanu1 (Bhanu)]

2020-04-01T05:45:22+00:00 log-forwarder-rs6zr loopbackappwithad-65f86ccd8d-gqw {"log":"2020-04-01T05:45:22.153Z\u0009INFO\u0009[monitoring]\u0009log/log.go:144\u0009Non-zero metrics in the last 30s\u0009{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70510,"time":{"ms":3}},"total":{"ticks":668260,"time":{"ms":38},"value":668260},"user":{"ticks":597750,"time":{"ms":35}}},"handles":{"limit":{"hard":1000000,"soft":1000000},"open":7},"info":{"ephemeral_id":"d82781ad-44f4-4733-ab0a-863190628e76","uptime":{"ms":401760599}},"memstats":{"gc_next":1244196,"memory_alloc":1130688,"memory_total":24228459296,"rss":69632}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.24,"15":0.2,"5":0.15,"norm":{"1":0.12,"15":0.1,"5":0.075}}}}}}\n","stream":"stderr","time":"2020-04-01T05:45:22.154078202Z"}