Extract log patterns

I am a past life SUMO logic user.
There are times when you want to analyze the log and extract / identify patterns rather than manually identifying them.
I am looking for something similar to the SUMO logic log reduce operator: https://help.sumologic.com/05Search/LogReduce/Detect-Patterns-with-LogReduce.

I have searched the ELK documentation and have not been able to find anything that matches that functionality.

Any help would be appreciated.

PS: Splunk has the pattern job functionality

In the Elastic Stack, pattern extraction happens before sending data to Elasticsearch and viewing in Kibana. It's handled by Logstash with grok, or by the ingest API in Elasticsearch with grok.

I think I did not state my problem right.

We do use grok patterns to extract messages and store it in elastic.

For example my application logs something like:
"Saved record with id: 1234"
"Saved record with id: 9870"
"Saved record with id: 4523"

it also logs
"Error: connection timeout"
"Error: database restarted"

So I have all these logs.

What SUMO and Splunk offer is way to extract these patterns from the messages and give a tabular output something to the following effect

Message pattern Frequency
Saved record with id: <> . 80%
Error: connection timeout 10%
Error: database restarted 10%

If this can be achieved using grok patterns, I would be interested to understand it.

What does your config look like?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.