# Kibana Error Alerting - Filter Similiar Error Messages | Similarity Query \[Elastalert\] **URL:** \ **Category:** Kibana\ **Created:** [January 28, 2021, 2:33pm UTC](https://discuss.elastic.co/t/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elastalert/262517 "2021-01-28T14:33:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![MartinJaskulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinjaskulla/32/83063_2.png) [@MartinJaskulla](https://discuss.elastic.co/u/MartinJaskulla)\ **Post date:** [January 28, 2021, 2:33pm UTC](https://discuss.elastic.co/t/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elastalert/262517/1 "2021-01-28T14:33:39Z")
My goal is to be alerted for unique error _types_ only (Elasticsearch 6.8 + [Elastalert](https://github.com/Yelp/elastalert)). Example error logs (5 documents, same index, same field): - Error 9837 in Component X trying to reconnect in 9.3 seconds - Error 9837 in Component X trying to reconnect in 8.7 seconds - Attempt to connect to 19.324.21.234 failed - 2021/01/01 08:51:54.203 Errno 111 tensorflow/stream\_executor... - 2021/01/01 08:52:76.009 Errno 111 tensorflow/stream\_executor... Using `query_key` from [Elastalert](https://elastalert.readthedocs.io/en/latest/ruletypes.html#query-key) I can make sure to be alerted for unique error messages only. The error messages above are all unique and therefore I will receive 5 alerts. However I only want to receive 3 alerts, one per error _type_ . Elastalert allows you to write regular queries with Elasticsearch's Query DSL. **Is there a way to write a query which filters out documents if a specific field is 100% - 90% similiar to other documents?** I suspect there might be a totally different solution to this. If there are better ways to handle alerting or some best practices, feel free to share them. ---
**Author:** ![wylie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wylie/32/81794_2.png) [@wylie](https://discuss.elastic.co/u/wylie)\ **Post date:** [January 28, 2021, 6:52pm UTC](https://discuss.elastic.co/t/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elastalert/262517/2 "2021-01-28T18:52:55Z")
Hi, welcome to the forums! Even though we don't offer help with non-Elastic products like Elastalert in the forums, I think your question can be answered in a simple enough way. I'm moving your post to the Elasticsearch section of the forum because you're asking for help with data modeling and queries. 1. The thing you're asking for is not possible using Elasticsearch _only_, you will need to add extra logic somewhere outside of Elasticsearch to do this. 2. Can you extract the "error type" into a separate field in your documents, which will be the unique key? This is most commonly done at ingestion time, such as using Logstash or an [ingest node](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html). Elasticsearch is best with semi-structured data, not fully unstructured data. ---
**Author:** ![MartinJaskulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinjaskulla/32/83063_2.png) [@MartinJaskulla](https://discuss.elastic.co/u/MartinJaskulla)\ **Post date:** [January 29, 2021, 2:02pm UTC](https://discuss.elastic.co/t/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elastalert/262517/3 "2021-01-29T14:02:09Z")
Answering my own question: We added a new field with the [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2021, 2:02pm UTC](https://discuss.elastic.co/t/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elastalert/262517/4 "2021-02-26T14:02:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.