Kibana: can I store "Time" as a variable and run a consecutive search?

[Francesco_Mantovani]

I'm new to Kibana and I'm trying to automate the research we do every day.
I already asked a question on StackOverflow but I'm not satisfied with the reply.

I work in a company that produce a software for autonomous vehicles. When a booking is rejected we need to understand why.
I need to monitor just a few seconds of logs on 3 different machines. Each log is completely separated, there is no relation between the logs so I cannot write a query in discover, I need to run 3 separated queries.

EXAMPLE:

1. A booking was rejected, so I open Chrome and I search on "elk-prod.myhost.com" for the BookingID: "b2c729b5-6440-4829-8562-abd81991e2a0" and I have a dozen of logs returned during a range of 2 seconds (FROM: September 3rd 2019, 21:28:22.155 , TO: September 3rd 2019, 21:28:23.524 ).

2. Now I need to know what was happening on the car so I open a new Chrome tab and I search on "elk-prod.myhost.com" for the CarID: "Tesla-45-OU" on the time range FROM: September 3rd 2019, 21:28:22.155 , TO: September 3rd 2019, 21:28:23.524

3. Now I need to know why the server which calculate the matching rejected the booking so I open a new Chrome tab and I search for the word CalculationMatrix always on the time range FROM: September 3rd 2019, 21:28:22.155 , TO: September 3rd 2019, 21:28:23.524

CONCLUSION: I want to stop to keep opening Chrome tabs by hand and automate the whole thing. I have no idea around what time the book was made so I first need to search for the BookingID "b2c729b5-6440-4829-8562-abd81991e2a0" , then store the timestamp of first and last log and run a second and third query based on those timestamps.

There is no relation between the 3 logs I search so there is no way to filter from the Discover, I need to automate 3 different query.

[mattkime (Matthew Kime)]

Have you considered using a single index pattern to cover all three log sources? You would still need a way to filter within the combined log set but you could use saved filters from a single discover window.

[Francesco_Mantovani]

I cannot use a saved filter.

I can use a saved filter maybe for the first search, where I can put {BookingID} and search for, let's say the last 7 days. That is unique, it make sense.

But then the second search need to search for {CarID} ... in what lapse of time?... in behalf of the laps of time of the previous search. I cannot search for {CarID} for the last 7 days, I will have tons of logs! I need to search for that {CarID} for the 2 - 3 seconds where the event happened. Only the first search can give me that time range.

The same for the 3rd search, I need to search for {CalculationMatrix} but once again for what time? One Week? One Month? No only the very tiny gap of time in which this event happened because the server is calculating that {CalculationMatrix} every 5 seconds and I need to search for a very specific range of time.

That's why I keep opening tabs and making new Discovery search.
And {BookingID}, {CarID} and {CalculationMatrix} they don't have a unique key that would link them all. They are completely unlinked.

But isn't this the ultimate goal of non relational databases, not having a primary key?

I cannot be the only one facing this sort of issue.

Thank you for your help BTW

[mattkime (Matthew Kime)]

Generally speaking, actions this particular might be worth writing code either as a kibana plugin or a script which directly queries elasticsearch.

in what lapse of time?... in behalf of the laps of time of the previous search.

Discover provides a time range selection which could be applied to all the relevant documents.

Maybe things could work this way - You search for the BookingID (across all 3 indices), then you search for the carID with an OR clause, and finally the calculation matrix would already be limited by time but would need another OR clause on a field only in that index.

I'm not entirely understanding the flow of looking up information (where do you find the carID? I'm trying to think of this in terms of inputs and outputs) but the above paragraph might provide another perspective. If its not useful we can go into more detail about how you're assembling the 3 queries.

[Francesco_Mantovani]

Thank you @mattkime,

I like writing code.

As I'm new to Kibana can you just tell me if there is a page where I can paste my code or if that is something I need to change in ELK under the hood?
I'm trying to solve the problem by myself without distracting Engineers.

I see there is Dev Tools, can I structure my code there?

[mattkime (Matthew Kime)]

Dev Tools is useful for running queries against elasticsearch. You can write more than one query and execute them one at a time. If you can write your queries in a useful way it might help you avoid switching windows. I don't think Dev Tools saves the contents so you'd want to copy your queries to a local text file. I think this would be a good starting point as it would be the basis for any more complicated solution.