Grok pattern to match different log formats in same log file

[Shweta_Priyadarshani]

Hi Everyone,

I am trying to apply filter in logstash conf file by giving grok pattern . But the challenge here is that my log file thats the input for logstash , has different formats. Ex:

1.) 27.06.2018 00:00:00.009 INFO [sling-default-5236-com.adobe.granite.threaddump.impl.BackupCleaner] com.adobe.granite.threaddump.impl.BackupCleaner File /AEM/prod/primary/author/crx-quickstart/threaddumps/20180620/threaddump.124633.txt.gz successfully deleted

2.) 27.06.2018 00:35:58.952 INFO [172.31.87.111 [1530059758951] GET /libs/granite/core/content/login.html HTTP/1.1] com.merckgroup.aem.healthcare.biopharma.neurology.merckneurology.filters.ValidateLicenseFilter path information/libs/granite/core/content/login.html

What can be done so that i can put grok pattern for all the different log formats .

Also , is there a way to determine how many different log formats i have in my log file , because my log file is more than 1lac lines so looking for different formats manually is a very tough job

Thanks in advance

[Badger]

The grok filter can take an array of patterns to match against. Be sure to anchor your patterns.

If you need many different patterns you are going to have to construct those patterns. There is no tool to do it for you. Take a log, index it into a disposable index, go see which events have a _grokparsefailure tag and construct patterns that match them. Delete the index and do it again.