Show top 100 most occured errors from log

[5JrA2ngS8Snub]

FYI: I'm new to using the ELK stack.

I want to have more (easier) insight into the apache error log data my servers are producing. My plan is to create dashboards that will show me the information I need. One of the most important things I want to know is: what is the most common error in my application for a period of X hours.

So I created a test environment using elasticsearch, logstash and filebeat. I have data coming in from a test application. This data is coming from a apache error log that is send to logstash using filebeat and the GROK extension.

I need some hints on how to create such top 100 error, I assume I have to do some kind of aggregation on the message field. But is this something I need to implement into the logstash or filebeat config or just in the Kibana interface when creating a dashboard?

Any advice to get me started would be appreciated

A simple representation of the output I would like to have in my dashboard:

#      Count:                   Error:

1      400                       'Unable to call undefined function testx'
2      20                        'Var X cannot be NULL'
...    ...                      ....

[matw (Matthias Wilhelm)]

Hi and welcome to our community!

You can certainly do this in Kibana, here's a similar case showing e.g. top referer in Kibana Lens

Similar stuff you can do with error messages. Takes 2 few clicks to build such a table there. Dragging the fields to the panel, choosing Data Table as Visualization, ready.

Best,
Matthias

[wylie (Wylie Conlon)]

One thing that's important is that you need to be able to aggregate on the field. Typically message fields are indexed as full text, but you will need to use a keyword field if you want to aggregate. There is more detail about this in the Elasticsearch text field docs