Extract log patterns

Elastic Stack Kibana
1

I am a past life SUMO logic user.
There are times when you want to analyze the log and extract / identify patterns rather than manually identifying them.
I am looking for something similar to the SUMO logic log reduce operator: https://help.sumologic.com/05Search/LogReduce/Detect-Patterns-with-LogReduce.

I have searched the ELK documentation and have not been able to find anything that matches that functionality.

Any help would be appreciated.

PS: Splunk has the pattern job functionality

2

In the Elastic Stack, pattern extraction happens before sending data to Elasticsearch and viewing in Kibana. It's handled by Logstash with grok, or by the ingest API in Elasticsearch with grok.

3

I think I did not state my problem right.

We do use grok patterns to extract messages and store it in elastic.

For example my application logs something like:
"Saved record with id: 1234"
"Saved record with id: 9870"
"Saved record with id: 4523"

it also logs
"Error: connection timeout"
"Error: database restarted"

So I have all these logs.

What SUMO and Splunk offer is way to extract these patterns from the messages and give a tabular output something to the following effect

Message pattern Frequency
Saved record with id: <> . 80%
Error: connection timeout 10%
Error: database restarted 10%

If this can be achieved using grok patterns, I would be interested to understand it.

4

What does your config look like?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.