Parse text to Json in Filebeat

[Bhanu1 (Bhanu)]

Need help to parse rsyslog data to elastic search

2020-04-01T06:12:05+00:00 log-forwarder-rs6zr myrtfapp-685c9695fd-pppnk_fe614c {"log":"2020-04-01T06:12:05.453Z\u0009INFO\u0009[monitoring]\u0009log/log.go:144\u0009Non-zero metrics in the last 30s\u0009{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":26350,"time":{"ms":5}},"total":{"ticks":247940,"time":{"ms":54},"value":247940},"user":{"ticks":221590,"time":{"ms":49}}},"handles":{"limit":{"hard":1000000,"soft":1000000},"open":7},"info":{"ephemeral_id":"426529a5-b992-47c8-be2f-a792fd82242b","uptime":{"ms":148590996}},"memstats":{"gc_next":1316312,"memory_alloc":1202712,"memory_total":8941460576,"rss":-241664}},"filebeat":{"events":{"added":51,"done":51},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":51,"batches":2,"total":51},"read":{"bytes":70},"write":{"bytes":3589}},"pipeline":{"clients":1,"events":{"active":0,"published":51,"total":51},"queue":{"acked":51}}},"registrar":{"states":{"current":1,"update":51},"writes":{"success":2,"total":2}},"system":{"load":{"1":0.01,"15":0.11,"5":0.05,"norm":{"1":0.005,"15":0.055,"5":0.025}}}}}}\n","stream":"stderr","time":"2020-04-01T06:12:05.454164098Z"}

[ChrsMark (Chris Mark)]

Hi!

Please provide more information about what is the output you are attaching. Is it from Filebeat?

In addition, regarding your case (rsyslog->Eleasticsearch) I would propose using the respective input plugin for Logstrash. I don't see how Filebeat could help in that case.

Regards

[Bhanu1 (Bhanu)]

Hi

Can u help me to get Grok pattern to parse Json inside the message

<13>Apr 2 13:18:07 log-forwarder-68kh5 kapacitor-7cf7f7bdd4-b86lb_monit {"log":"ts=2020-04-02T13:18:06.957Z lvl=info msg="http request" service=http host=127.0.0.1 username=- start=2020-04-02T13:18:06.953503769Z method=GET uri=/kapacitor/v1/tasks?dot-view=attributes\u0026fields=type\u0026fields=status\u0026fields=executing\u0026fields=dbrps\u0026limit=100\u0026offset=0\u0026pattern=\u0026replay-id=\u0026script-format=formatted protocol=HTTP/1.1 status=200 referer=- user-agent=KapacitorClient request-id=6421f945-74e4-11ea-93c7-000000000000 duration=3.647289ms\n","stream":"stderr","time":"2020-04-02T13:18:06.961284654Z"}

[ChrsMark (Chris Mark)]

@Bhanu1 Hi!

There are a lot of resources out there, I already have shared some with you, on how to create grok patterns so why you don't give it a shot?