To find all the correlation id where response is not send back to the consumer

[rohitarorait82 (Rohit arora)]

Hi All,

In our logs; For a transaction , two legs will get generated "Consumer to backend" and "Backend to Consumer" .However in case of failure , only one leg "Consumer to backend" will get generated.

Unique correlation id will get generated for each transaction consisting of these two legs.
Is there a way to find out all the correlation id for which second leg is not generated.

[mastersmit (Smit)]

you may use Logstash Aggregation to aggregate by correlationid.

And as many events you receive for that correlationid you may increase the count.

if the count is 2, you could know it was successful...

Just a quick suggestion...

[Mark_Harwood]

Sounds like a typical use case for transforms

[rohitarorait82 (Rohit arora)]

Thanks @Mark_Harwood and @mastersmit : I applied some filters and in the visualization, I can see for few correl_id I am getting count = 1 and for few I am getting count =2. Is there a way to display only where count = 1.

Something like min_doc_count , do we have max_doc_count in v7.7 ELK

[Mark_Harwood]

If you use the transforms API to create an entity-centric index with a single doc for correlation ID then you can have properties like count and duration as physical fields which you can sort and filter on.

[ylasri (Yassine LASRI)]

One quick solution that may help is to visualize using Canevas , that rely a lot on SQL so you can get quickly all id that occure only once

POST _sql?format=txt
{
  "query": """
  SELECT clientip, count(*) FROM kibana_sample_data_logs group by clientip having count(*) <= 5 order by 2 desc
  """