Parse log line into JSON even when some tags from log can miss

Elastic Stack Logstash
1

Hi,

My log line looks like below. In my log it is possible that few tags could miss (example: CITY or ZIPCODE or any other tag may not always be present). I want to parse this log and show output as JSON.

Log Line:
20180524T104101.550 GMT, ID=A-124, FIRST_NAME=ABC, LAST_NAME=XYZ, ADDRESS1=123 STREET, CITY=NEW YORK, STATE=NEW YORK ZIPCODE=40001

I tried giving regex patterns but they work only when all tags are present if any of the random tags are not present then I get _grokparsefailure

Can you please help me know if it is possible to parse log into JSON even when some tags can miss.

Thank you!

2

Use a grok filter to extract the timestamp into one field and the rest of the string into another field. Then feed the latter field to a kv filter.

3

Thanks. Let me try above suggestion.

4

It worked. Thank you very much. I want to remove default fields like host, message, @timestamp, @version, etc. Is there a way to do it?

5

You can't remove @timestamp but all other fields can be removed with a remove_field option in any existing filter or in a mutate filter of its own.