How to get the counts of log messages based on the error type in Elasticsearch visualization

[vampire (Ming Huang)]

I have some sample data on the Elasticsearch, which looks like the following:

I am using the data table in the Visualize section to get the counts for each error type, for example: it should output

Error: Update failed for online booking with id, count is 5.

Not the count 1 for different id of the same error type.

What I have done is to build a query to output the counts for each error type, which looks like this:

However, when I save the query as the saved search, then visualize it as data table, it still have the same issue as above.

I was thinking to only save the output of that query as saved search, one issue is that the output is too verbose, has a lot of information I don't really need.

Any suggestions please !

[ppisljar (Peter Pisljar)]

could you show a sample of your document ?

[vampire (Ming Huang)]

I am sorry that I cannot show some sample. my concern is that I can create a query to get the counts for each error message using regex, however, in terms of visualization, we have no way to get the same result.

[Babadofar (Christoffer Vig)]

You should preprocessor your documents, logstash can do some for you, like remove numbers from Messages etc. Also, create visualisation in kibana, first, it will generate query

[vampire (Ming Huang)]

Thanks for the reply, Christoffer. However, the real data is much more complicated than this. so simply remove the number is not gonna work. I am thinking about using the scripted fields. do you have any suggestion doing that?

[Babadofar (Christoffer Vig)]

My suggestion is to pre process data, before it reaches elasticsearch. Kibana scripted fields are applied on the fly when executing query, and is not very powerful. Look into logstash filters. https://www.elastic.co/guide/en/logstash/current/filter-plugins.html and elasticsearch analysers https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.htm
If logstash does not have the filter you need, then you can create a custom application to transform your data.
Analysers are a key ingredient in making complex data searchable, which you need to use even if Logstash doesn't cut the mustard.