How to group application logs by error uniqueness and count occurrences in Elasticsearch/Kibana?

Elastic Stack Logstash
1

Hi everyone,

We are currently shipping our application logs directly to Elasticsearch, and we are trying to find a way to analyze our error patterns more effectively. Specifically, we need to:

  1. Identify the uniqueness of our errors (grouping identical types of errors together).

  2. Find out the exact occurrence count for each unique error type.

The Problem: If we do a standard terms aggregation on our raw message or text fields, every log line looks completely unique. This is because our log strings contain highly dynamic variables that change on every request, such as:

  • User/Tenant IDs (e.g., UUIDs)

  • Dynamic database/row numeric IDs

  • IP addresses and hostnames

  • Hex strings or timestamps embedded in the message

Because of this high cardinality, our dashboards are flooded with thousands of individual buckets instead of showing us the top 5 or 10 structural errors that are actually breaking our application.

Our Goal: We want a clean way to strip away or ignore these dynamic variables so Elasticsearch can recognize that Connection timed out to database-123 and Connection timed out to database-456 are actually the exact same error, and count them together as 2 occurrences.

My questions for the community:

  1. What is the standard industry practice or architecture for grouping unstructured logs by error patterns inside the Elastic Stack?

  2. Should this deduplication/normalization be handled during ingestion (via an Ingest Pipeline or Logstash), or is there a way to do this at query time/inside Kibana?

  3. Are there any native features (like Machine Learning or out-of-the-box processors) that handle this automatically without requiring us to manually maintain a massive list of regex patterns?

2

Identify the uniqueness of our errors (grouping identical types of errors together).
-> If errors are from logs, parse the line, show in the Kibana dashboard or analitic by type, error code or similar
-> If errors are during log processing in LS, you can handle it and send to separate index and analyze only them

Find out the exact occurrence count for each unique error type.
-> Again, grouping by something in the Kibana dashboard or patterns?
-> Most likely you can use AI, if you have enterprise license

If you have unrelated error in log, go back to developers to improve logs. Can you imagine is web server have logs like you described? Error code, subcode... for HTTP is very helful.

  1. What is the standard industry practice or architecture for grouping unstructured logs by error patterns inside the Elastic Stack?
    -> If you have unstructured loga, maybe you can additionaly parse fields in ES QL by grok or dissect.

  2. Should this deduplication/normalization be handled during ingestion (via an Ingest Pipeline or Logstash), or is there a way to do this at query time/inside Kibana?
    -> For deduplication make your own _ids, described here.

  3. Are there any native features (like Machine Learning or out-of-the-box processors) that handle this automatically without requiring us to manually maintain a massive list of regex patterns?
    -> There is some a pattern analysis like this, however without more details I don't think that others from Elastic Jedi Order will have much idea. However, others will suggest something and ask more question if they need