Parsing log message optional fields

I'm supporting an application that generates log messages that contain 65 fields. More than 40 fields are optional that means sometimes according to application logic, optional fields exits on log message and sometimes not. I'm using grok filter plugin but the problem is if an optional filed appears on the log message that not match to grok pattern, the log message will not filter and will be bypass by logstash. And also if I write a grok pattern for all of the states, then I should write more 40! grok filter that impossible. What's the best solution for filtering optional fields on log message?

It would be a lot easier to answer this if you could give a couple of examples of the log messages.

Here is log message format:

MANDATORY|MANDATORY|MANDATORY|optinal|optional|MANDATORY|optional|MANDATORY|MANDATORY|MANDATORY|optional|MANDATORY|MANDATORY|optional|optional|MANDATORY|optional|optional|optional|optional|optional|optional|optional|MANDATORY|optional|optional|MANDATORY|MANDATORY|MANDATORY|MANDATORY|MANDATORY|optional|optional|optional|MANDATORY|optional|optional|optional|optional|optional|MANDATORY|MANDATORY |optional|optional|optional|optional|optional|optional|optional|MANDATORY|optional|optional|optional|MANDATORY|optional|optional|optional|optional|optional|optional|optional|optional|optional|optional|

And here is log message example:

432757578889|544646998109|1|||FSFSS_155793420754833334804|222222-1111-3333-a3699cddwaFFFFF0c11099|2019-07-15 20:00:07|654345424|d1ddfnk1a8f36566fefb9|FFFFDSAFESEFSEF|19545442|MSDFDSFSD||438434343|DSFDSF ||||||||hyygyiig|043|4340|DSFDSFWDF44455|de444dfs|15000000|54543340|51434314||33||DE|044|044|440||0|0|success ||35|53||1||eww|eGG|||0|75|1|1|2|86576786654||r444||5555|

The field descriptor is pipeline sign (|), and optional fields show . According to application logic, one or more that one optional filed will appear on log message and other optional filed that does not exist on log message will be shown as blank, for example below string show to optional filed that not exist in log message:


Why not use a csv filter?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.