Making a grok filed optional in grok pattern

[Pranjal_Sett]

Hi,

I am trying to make my field optional. Let me give you the scenario below:

let suppose my log patterns are:

pattern 1:

[2022-04-25T12:51:27.967+02:00] [d03-c02-s02] [NOTIFICATION] [oracle.wsm.msg.logging] [tid: 102] [userId: ] [ecid: *] [APP: soa-infra] [partition-name: DOMAIN] [tenant-name: GLOBAL] [WSM_ServiceID: default/SalesTransactionEBSV2/SalesTransactionMediator] [WSM_OperationName: *] [FlowId: xyz] [oracle.wsm.policy.name: oracle/log_policy] [WSM_LogType: Request] [[XML]]

pattern 2:

[2022-04-22T13:11:14.858+02:00] [d03-c02-s02] [NOTIFICATION] [oracle.wsm.msg.logging] [tid: 137] [userId: ] [ecid: *] [APP: soa-infra] [partition-name: DOMAIN] [tenant-name: GLOBAL] [WSM_ServiceID: *] [FlowId: abc] [oracle.wsm.policy.name: oracle/log_policy] [WSM_LogType: Request] [[XML]]

pattern 3:

[2022-04-08T19:40:11.536+02:00] [d03-c02-s02] [NOTIFICATION] [oracle.wsm.msg.logging] [tid: 2729] [userId: ] [ecid: *] [APP: soa-infra] [partition-name: DOMAIN] [tenant-name: GLOBAL] [J2EE_APP.name: soa-infra] [J2EE_MODULE.name: fabric] [WEBSERVICE.name: *] [WEBSERVICE_PORT.name: *] [WSM_RemoteAddress: *] [WSM_ServiceID: *] [WSM_OperationName: *] [oracle.wsm.policy.name: oracle/log_policy] [WSM_LogType: Response] [[XML]]

So if you noticed here upto tenant-name all of them is having same patterns but after that it was changed some times.

My question is how can I make all those fields optional after tenant-name. I have tried some of the solutions but it got fails.

tried this solution Making a part in the grok expression optional , but I feel my "[" this bracket creating some problem here! Not able to match grok patterns.

Thanks,
Pranjal

[Pranjal_Sett]

@Badger , Can you please help on this?

[Badger]

@Pranjal_Sett

[J2EE_APP.name: soa-infra]

If that field is optional then you can use ? (zero or more) or {0,1} (zero or exactly one) to indicate that. The square brackets must be escaped to show that they are not part of a character group. Use parentheses to indicate what {0,1} applies to

(\[J2EE_APP.name: [-\w]+\]){0,1}

In some cases it is easier to search for multiple patterns.