Display non Aggregated field on Y-axis with date Histogram


(pankaj bhagra) #1

I am impressed with the capabilities of ELK stack for meaningful data extraction and visualization. I am able to do lots of interesting stuff with it, but hitting a wall for a very basic kind of usecase.

I would like to plot the state transition of a given numeric field on y-axis with date histogram in x-axis. The interest is to visualize the edge transaction of the given field. this would be say timePlot of state machine making migration b/w fixed number of states (say under 6 number of states). The timestamp are in millisec granularity. I am not interested in the aggregated state change count (nor min/max), but interested in plotting how the state is changing with time (simple xl type of graph).

Any suggestions/recommendation/pointers how to achieve this in ELK ?

later,
pankaj


(Tanya Bragin) #2

Could you share a couple of sample documents you are working with to better illustrate your use case?


(pankaj bhagra) #3

Thanks Tanya for picking my query. My data looks like below, for which i use logstash to insert in the elastic search. The timeStamp (time/date field) and Event code (integer) are available to kibana. Now i want to draw a time histogram (line chart) with Event code on Y-axis. Is it doable in Kibana (no aggregation) or in backend EL(k)? If not, what would be next best bet ?

12.05.2015 15:17:36.802 Event 2 (Motion)
12.05.2015 15:17:38.732 Event 0 (Rest)
12.05.2015 15:17:38.742 Event 2 (Motion)
12.05.2015 15:17:41.872 Event 3 (TurboPace)
12.05.2015 15:17:41.992 Event 2 (Motion)
12.05.2015 15:17:46.902 Event 3 (TurboPace)
12.05.2015 15:17:47.032 Event 2 (Motion)
12.05.2015 15:17:48.632 Event 1 (SlowingDown)
12.05.2015 15:17:48.750 Event 0 (Rest)


(Tanya Bragin) #4

Unfortunately, I don't think it is possible to simply plot that data, because on the X-Axis you do have to bucket by date, at which point you may end up with more than one data point in the bucket. The smallest bucket possible is 1s, so your second and third lines in your example will end up on one bucket. At that point, Y-Axis has to perform some function over the values in the bucket. You can "count" them, "average" over them, pick "max" or "min" and so on.

What is the ultimate goal you're trying to achieve with this visualization?


(pankaj bhagra) #5
  1. what if i don't want to bucketize the data and plot them as independent entities on the time graph (connecting dots linearly in time graph). I guess its not possible, but wanted to ask to make sure.
  2. u mentioned that smallest unit for bucketing is 1sec. but somewhere during my search/reading, i came across some references of adding custom units (partial and millisec). Is it not possible.
  3. The usage is pure for visual interpretation of data by end user.

(John Sotiropoulos) #6

Hi !

Did you find any solution for your problem? If yes, could you please describe what you did? Did you make any use of the scripted fields or json inputs?

I want to do something more complex but in the end I want to combine some calculations and plot them over time. The biggest constraint here is that in the Y-axis you have to choose between Count, Min, Max, etc and not a specific field. Have you found any workaround?


(system) #7