Join documents in ELK

Hello Everyone,

Hope everyone is doing great and staying healthy.

As i'm new to ELK, I need input/suggestions from the ELK experts on following use case that i'm currently working on.

I'm working on a dashboard to monitor my batch processes. I wanted to show in this dashboard current status of my batch, i.e. "what suppose to happen in a given day, what happened [completed], what is happening now [currently running] and what is yet to happen [Pending]" based on the logs.

We have metadata with the information what suppose to happen [i.e. Job details - Job Name that need to be run in a given day]. We are writing an event to ELK when each job start/finish [sending Job Name, Job start/end time, file name, records processed, etc.. to ELK as part of the log]. Now i need to join these two documents [Metadata and Logs] and check what is completed, running and pending. Would it be possible to join two documents in Kibana and find out the status of each job? In relational database i can easily find it by doing left outer join between Metadata table and log table with following logic.

  • if the job name is present in log table with completed status then that job is completed
  • if the job name is present in log table with running status then that job is running
  • if the job name is not present in log table then that job is not yet started

Would it be possible to implement the similar logic in ELK? If yes, can you please provide me few ideas on how to implement this?

Appreciate your help!

Thanks,
Sri

In a Kibana table visualization:

  • Use bucket terms aggregation for the job name.
  • Use Top Hit metric aggregation for the status field and set size=1 and sort it by @timestamp to get most recent status.

Thank you @Alsheh for your inputs.

This will give the information of jobs completed and currently running but not the pending because there are no logs for the pending jobs yet. For this to know, i need to check the logs against my jobs metadata to find out what is pending. Would it be possible to compare the data returned after aggregating logs data [find out the latest job status for each job] against other document [in this case jobs metadata] in ELK?

I've different categories in my batch process and for each category i've start and end time that need to be used to check the logs received in this period. For example if i've 2 categories, Category 1 has 10 jobs and Category 2 has 15 jobs. For category 1, need to check the logs received between 6 AM - 1 PM and for Category 2, check the logs received between 11 AM - 5 PM and report the status.

Thanks,
Sri

Hello All,

Can anyone provide suggestions/recommendations on how to do the above in ELK?

Thanks,
Sri

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.