ElasticSearch Grok Filter

Hi Everyone, I try to apply grok filter for my logs. As you see I am getting some of them. But I don't get first.jobId=0 or first.applicationname=POLY2

How can I get these fields?

Here is my log

2022-06-24 18:43:03,628 HM_PROFILER INFO [lwolpert] [BBy7Twm70pt07CFfSPDjoCILcKsBJV3d7TkAyfWy] |firs.jobId=0|firs.applicationname=POLY2|firs.method=getExaminerScheduled|firs.class=gov.dhs.usss.pars.database.AgentDbUtil|firs.action=|firs.entity=|firs.entityAction=|firs.timeMetric=7|firs.applicationLayer=DATABASE|

Hello @HKN_MZ

Try this below grok Pattern which would get jobid and appid

%{DATESTAMP:Timestamp} %{WORD:application} %{WORD:log_type} \[%{WORD:user}\] \[%{WORD:id}\] \|%{DATA:jobid}\|%{DATA:appid}\|%{GREEDYDATA:unwanted}

Keep Posted !!! Thanks !!! :smiley:

For this type of data I would recommend first separating out the string containing the key-value pairs using grok and then applying a KV processor. The KV processor will handle changes to the order of the keys and should be more efficient than grok here.

Thanks for your repplies,
Here is the my way,

%{DATESTAMP:Timestamp} %{WORD:application} %{WORD:log.type} \[%{WORD:user}\] \[%{WORD:id}\] \|%{DATA:UNWANTED}\=%{USERNAME:firs.jobId}\|%{DATA:UNWANTED}\=%{USERNAME:firs.applicationname}\|%{DATA:UNWANTED}\=%{DATA:firs.method}\|%{DATA:UNWANTED}\=%{DATA:firs.class}\|%{DATA:UNWANTED}\=%{DATA:firs.action}\|%{DATA:UNWANTED}\=%{DATA:firs.entityAction}\|%{DATA:UNWANTED}\=%{USERNAME:firs.timeMetric}\|%{DATA:UNWANTED}\=%{USERNAME:firs.applicationLayer}

Output is here.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.