Parse log line into JSON even when some tags from log can miss


(Vinit) #1

Hi,

My log line looks like below. In my log it is possible that few tags could miss (example: CITY or ZIPCODE or any other tag may not always be present). I want to parse this log and show output as JSON.

Log Line:
20180524T104101.550 GMT, ID=A-124, FIRST_NAME=ABC, LAST_NAME=XYZ, ADDRESS1=123 STREET, CITY=NEW YORK, STATE=NEW YORK ZIPCODE=40001

I tried giving regex patterns but they work only when all tags are present if any of the random tags are not present then I get _grokparsefailure

Can you please help me know if it is possible to parse log into JSON even when some tags can miss.

Thank you!


(Magnus Bäck) #2

Use a grok filter to extract the timestamp into one field and the rest of the string into another field. Then feed the latter field to a kv filter.


(Vinit) #3

Thanks. Let me try above suggestion.


(Vinit) #4

It worked. Thank you very much. I want to remove default fields like host, message, @timestamp, @version, etc. Is there a way to do it?


(Magnus Bäck) #5

You can't remove @timestamp but all other fields can be removed with a remove_field option in any existing filter or in a mutate filter of its own.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.