Hope everyone is doing great and staying healthy.
As i'm new to ELK, I need input/suggestions from the ELK experts on following use case that i'm currently working on.
I'm working on a dashboard to monitor my batch processes. I wanted to show in this dashboard current status of my batch, i.e. "what suppose to happen in a given day, what happened [completed], what is happening now [currently running] and what is yet to happen [Pending]" based on the logs.
We have metadata with the information what suppose to happen [i.e. Job details - Job Name that need to be run in a given day]. We are writing an event to ELK when each job start/finish [sending Job Name, Job start/end time, file name, records processed, etc.. to ELK as part of the log]. Now i need to join these two documents [Metadata and Logs] and check what is completed, running and pending. Would it be possible to join two documents in Kibana and find out the status of each job? In relational database i can easily find it by doing left outer join between Metadata table and log table with following logic.
- if the job name is present in log table with completed status then that job is completed
- if the job name is present in log table with running status then that job is running
- if the job name is not present in log table then that job is not yet started
Would it be possible to implement the similar logic in ELK? If yes, can you please provide me few ideas on how to implement this?
Appreciate your help!