Kibana Error Alerting - Filter Similiar Error Messages | Similarity Query [Elastalert]

[MartinJaskulla]

My goal is to be alerted for unique error types only (Elasticsearch 6.8 + Elastalert).

Example error logs (5 documents, same index, same field):

• Error 9837 in Component X trying to reconnect in 9.3 seconds
• Error 9837 in Component X trying to reconnect in 8.7 seconds
• Attempt to connect to 19.324.21.234 failed
• 2021/01/01 08:51:54.203 Errno 111 tensorflow/stream_executor...
• 2021/01/01 08:52:76.009 Errno 111 tensorflow/stream_executor...

Using query_key from Elastalert I can make sure to be alerted for unique error messages only. The error messages above are all unique and therefore I will receive 5 alerts. However I only want to receive 3 alerts, one per error type .

Elastalert allows you to write regular queries with Elasticsearch's Query DSL.

Is there a way to write a query which filters out documents if a specific field is 100% - 90% similiar to other documents?

I suspect there might be a totally different solution to this. If there are better ways to handle alerting or some best practices, feel free to share them.

[wylie (Wylie Conlon)]

Hi, welcome to the forums! Even though we don't offer help with non-Elastic products like Elastalert in the forums, I think your question can be answered in a simple enough way. I'm moving your post to the Elasticsearch section of the forum because you're asking for help with data modeling and queries.

1. The thing you're asking for is not possible using Elasticsearch only, you will need to add extra logic somewhere outside of Elasticsearch to do this.

2. Can you extract the "error type" into a separate field in your documents, which will be the unique key? This is most commonly done at ingestion time, such as using Logstash or an ingest node. Elasticsearch is best with semi-structured data, not fully unstructured data.

[MartinJaskulla]

Answering my own question:

We added a new field with the Levenshtein distance