I have a set of hierarchical jobs which are executed every night as part of a batch. I have log files where each line contains the name of the job prefixed with the names of all its parent jobs up to the highest level, time take by the job and the date on which the batch was executed.
I parsed the log files through logstash and fed the output to elasticsearch, by creating a custom index. Each document in the index consists the following fields:
Each job may have zero or more parents depending on its position in the hierarchy. There is an additional document that represents the batch and the time taken by the batch as a whole.
I add new documents to the same index every day.
I have created a Kibana bar chart visualization with max aggregation for job time on y axis and date histogram aggregation for the batch date on x axis.
Now, I need to compare the dates on which the batch took the maximum and the minimum time. I then need to figure out which job caused the most difference in the batch times of the two dates.
Is there any way to query the dates on which the job took the max and the min times? If so, is there a way to use its result to compute the difference between the job times of each job for both dates and find out the job that causes the max time difference.
I'm using elasticsearch 5.2.0 and kibana 5.2.0 on windows 7.
Thanks in advance