How to find difference in ingested logs using Kibana visualization?

Elastic Stack Kibana
1

I have a continuous stream of logs being ingested in ELK stack and created visualizations in Kibana view for monitoring purposes. The logs are ingested from an server application which receives HTTP requests from client.

One of the new monitoring requirement is to find out requests failed at time T1 is succeeded at time T2 or not. In the logs, we have timestamp, request ID and request status.

Example:

Failure request log

{
  "time_start": "10/Apr/2020:16:36:05 +0000",
  "status": 500,
  "request_id": "7974457c",
  "object_name": "fileA.txt",
}

Success request log

{
  "time_start": "10/Apr/2020:16:56:59 +0000",
  "status": 200,
  "request_id": "3e35bd25",
}

As seen here, there was a retry for the same object 20 minutes later. Using Kibana visualization I want to know if there was an object retry from client performed or not. As seen in the below table, column A have failed object names, while column B have succeeded object names. File4 and File8 object were not retried by the client. enter image description here

Using Kibana, how can I find out such difference?
Right now I'm using a script that queries the data node using the index.
Can it be achieved using a plugin?

2

Because you are doing matching across multiple documents, which is a type of join, you need to do some custom scripting. It could be done in a Kibana plugin because you can make multiple requests and join the results.

3

That gives us a confidence to start looking at the Kibana plugins.
As per my understanding, here could be the flow for me - develop Kibana plugin, deploy, and use it to visualize the data I want.

Correct me if I'm wrong or bless me.

Thank you.

4

The only other option that might exist for you is to build a custom Vega visualization: this is not easy to do, but theoretically possible.

5

Wylie, I'm still a noob in ELK ecosystem, but can you tell me which one is easy and feasible solution to my question? - Kibana plugin or vega visualization?

6

I don't think the thing you are trying to do will be easy with either approach. You will need to decide whether you want to learn how to use Vega or learn the Kibana plugin system.

7

Wylie, are there any examples to look at that might help me?