ElasticSearch Grok Filter

[HKN_MZ]

Hi Everyone, I try to apply grok filter for my logs. As you see I am getting some of them. But I don't get first.jobId=0 or first.applicationname=POLY2

How can I get these fields?

[HKN_MZ]

Here is my log

Blockquote
2022-06-24 18:43:03,628 HM_PROFILER INFO [lwolpert] [BBy7Twm70pt07CFfSPDjoCILcKsBJV3d7TkAyfWy] |firs.jobId=0|firs.applicationname=POLY2|firs.method=getExaminerScheduled|firs.class=gov.dhs.usss.pars.database.AgentDbUtil|firs.action=|firs.entity=|firs.entityAction=|firs.timeMetric=7|firs.applicationLayer=DATABASE|

[sudhagar_ramesh]

Hello @HKN_MZ

Try this below grok Pattern which would get jobid and appid

%{DATESTAMP:Timestamp} %{WORD:application} %{WORD:log_type} \[%{WORD:user}\] \[%{WORD:id}\] \|%{DATA:jobid}\|%{DATA:appid}\|%{GREEDYDATA:unwanted}

Keep Posted !!! Thanks !!!

[Christian_Dahlqvist]

For this type of data I would recommend first separating out the string containing the key-value pairs using grok and then applying a KV processor. The KV processor will handle changes to the order of the keys and should be more efficient than grok here.

[HKN_MZ]

Thanks for your repplies,
Here is the my way,

%{DATESTAMP:Timestamp} %{WORD:application} %{WORD:log.type} \[%{WORD:user}\] \[%{WORD:id}\] \|%{DATA:UNWANTED}\=%{USERNAME:firs.jobId}\|%{DATA:UNWANTED}\=%{USERNAME:firs.applicationname}\|%{DATA:UNWANTED}\=%{DATA:firs.method}\|%{DATA:UNWANTED}\=%{DATA:firs.class}\|%{DATA:UNWANTED}\=%{DATA:firs.action}\|%{DATA:UNWANTED}\=%{DATA:firs.entityAction}\|%{DATA:UNWANTED}\=%{USERNAME:firs.timeMetric}\|%{DATA:UNWANTED}\=%{USERNAME:firs.applicationLayer}

Output is here.