How to index json attributes on ES from a json multiline log

Elastic Stack Beats
,
1

Hi All,

I need some advice here, I don't want to reinvent the wheel.
With Filebeat 6.7.0 I'm picking up logs in typical docker format, , i.e. json strings with "log", "stream" and "time" attributes.
In the logs, each event is actually split in multiple lines of the same format, that I re-assemble back using multiline.pattern in Filebeat..
So for example, in the logs I find 1 event split in these lines (I added spaces to show the log-stream-time format):

{"log":"2020-12-08T10:36:03.621Z - info: Request \n",           "stream":"stdout", "time":"2020-12-08T10:36:03.621670333Z"}
{"log":"{ attr01: 'value01',\n",                                "stream":"stdout", "time":"2020-12-08T10:36:03.621687063Z"}
{"log":"  attr02: 'value02',\n",                                "stream":"stdout", "time":"2020-12-08T10:36:03.621691804Z"}
{"log":"  attr03: 'vslue03',\n",                                "stream":"stdout", "time":"2020-12-08T10:36:03.621695644Z"}
{"log":"  attr04: { attr05: 'value05', attr06: 'value06' },\n", "stream":"stdout", "time":"2020-12-08T10:36:03.621702493Z"}
{"log":"  attr07: { attr08: 'value08', attr09: 'value09' },\n", "stream":"stdout", "time":"2020-12-08T10:36:03.621702493Z"}
{"log":"  attr10:\n",                                           "stream":"stdout", "time":"2020-12-08T10:36:03.621707294Z"}
{"log":"   { attr11: 'value11',\n",                             "stream":"stdout", "time":"2020-12-08T10:36:03.621721053Z"}
{"log":"     attr12: 'value12',\n",                             "stream":"stdout", "time":"2020-12-08T10:36:03.621724964Z"}
{"log":"     attr13: 'value13' } }\n",                          "stream":"stdout", "time":"2020-12-08T10:36:03.621739265Z"}

And thanks to multiline.pattern, Filebeat reassembles the "log" values, giving me 1 single event in ElasticSearch where "message" is a multiline string, like:

2020-12-08T10:36:03.621Z - info: Request
{ attr01: 'value01',
  attr02: 'value02',
  attr03: 'value03',
  attr04: { attr05: 'value05', attr06: 'value06' },
  attr07: { attr08: 'value08', attr09: 'value09' },
  attr10:
   { attr11: 'value11',
     attr12: 'value12',
     attr13: 'value13' } }

My question: I wish to see all those attr* in the json multiline string as searchable attributes, as now I have only a multline string in message.
Ideally in the ES event I wish to see attributes like:

....
message.attr01: 'value01'
message.attr02: 'value02'
....

Is it easy to achieve? Any hint is more than welcome

Cheers

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.