Timelion: Showing a drop in value as a point

[Nikhil_Utane]

Hi,

I am plotting device uptime which is shown correctly as seen below.

Is it possible to show the instance when the line drops as a single point?
For e.g. w.r.t to above chart, there should be two points at the twin peaks.

Right now I am plotting it as:
.es(metric='max:ruUPTime') (omitting other fields)

-Thanks
Nikhil

[timroes]

Hi Nikhil,

could you clarify a bit more how the chart should look? (maybe draw a quick version of it). I am not getting where exactly you want the points to be. Also I am not exactly sure how your documents look?
For which points in time in the above chart do you have documents in ES?

Could you perhaps add .points() to the expression and paste it here (it makes it more clear where your documents are).

Cheers,
Tim

[Nikhil_Utane]

Hi Tim,

Here's the points version of it.

I have marked the position in red where points should be shown.
The condition I want to apply is, show a point if the next value is lower than the previous value.
Hope that clarifies it.

-Thanks
Nikhil

[timroes]

Hey Nikhil,

this is actually kind of possible, by using the .derivative() function to detect the drop of the line.

The following example would draw a point at a fixed value (10) every time the line dropped:

.es(metric='max:ruUPTime').derivative().if(lt, 0, 10, null)

It first build the derivative - which will be negative everytime the line dropped and positive in all other cases. Then we'll use the if function on it, to set the value to 10, if the derivative was negative, and to null (i.e. remove it from the graph) if it was positive.

This is very very slightly different from what you asked, since this points will actually be not the one you marked red, but the first one after it dropped (since this is the data point, where the derivative got negative). Also it just places the point at a fixed height (10 or whatever you like).

If you want to show the point on the y-axis at the actual value (uptime) the server had before, you need a slightly more complex expression:

.es(metric='max:ruUPTime').derivative().if(lt, 0, .es(metric='max:ruUPTime', offset=-1s, fit=carry), null)

That way, if the derivative gets negative, you won't just fill in an static value, but you will fill in the value, that this series had 1 second earlier (offset=-1s), while using the carry fit method, meaning, it doesn't interpolate the value between the previous and the current value, but just leave it at it's previous value until it changes. Which in this case mean, the value 1 second earlier will still be the same as the previous data point.

I hope this expressions helped you getting started. Please feel free to reach out with any further questions

Cheers,
Tim

[Nikhil_Utane]

@timroes

.es(metric='max:ruUPTime').derivative().if(lt, 0, 10, null)

Thanks Tim. I figured I have to use derivative but wasn't clear how to show the point at the right place.
This is almost working. Only issue is, I don't see anything until I add a points() function. Also if I do offset of 1 sec, the y-axis value is different than if I do few hours. In this particular data set, there is a gap after the value drops, so looks to be the fit(carry) isn't working they way we want it to.

Any thoughts?

-Thanks & Regards
Nikhil

[Nikhil_Utane]

Here are the input events corresponding to the first red dot in the earlier post.

.

(1) With the above command as-is:

(2) After I add points() at the end:

(3) After i change the offset to -5h (hours). Anything less and it shows above chart (#2). This one correctly shows the Y-axis values.

Any idea what's going on here?

-Thanks
Nikhil

[Nikhil_Utane]

OK. Sort of understood what is going on. Since I am looking at larger time interval (last 60 days) each bucket in the date histogram is quite large (5-6 hours). So the derivative function goes to negative in one bucket and then back to positive in another (so about 10-12 hrs). Therefore the last negative derivative value is along the leading edge which corresponds to bucket with max RuUptime value of 43618. If we take the offset to > 5 hr, then we get the previous max value (and the right one) which is 1516873.

If I select a smaller, then the query works fine.
Any idea how to fix this?

-Thanks
Nikhil

[timroes]

First: sorry I forgot the points(). That will of course be needed, since you are working with very sparse data now.

Regarding the other problem. I think I understand the issue, and I guess the only solution for this is, using actual an offset, that matches your interval size (that you select beside the expression input). That way you should always get the previous bucket an thus the value from it.

I unfortunately can't think of a solution, where you find the real highest value, since that would require, we actually search through the individual documents and find the highest value, but that way of switching from aggregations to documents is not possible in Kibana. You would need to write your own custom application, that just communicates with the Elasticsearch API for that.

Cheers,
Tim