We are collecting some values of our servers, e.g. read and write to disk, using collectd + elk stack. The collecting part was no problem at all. Discovery is also fine, but now I want visualize the data.
The problem is in the aggregation part. Because I do not want to aggregate read/write of disk. I just want their value over a period, e.g. last 24 hours with hourly intervals. I have tried to find the result I need using Date Histogram on the X-axis but then what should I do on the Y-axis? Is this possible with Kibana 4; and if so, how should I approach it?
The Y axis you would change it from the default value of Count to something like MAX, AVERAGE, OR MIN and select the Numeric Field you want to view for reads. (On the Y axis you can add multiple values, by adding another aggregation.
I am not sure of the CollectD interface your using but if you do not see numberic fields in your available options to SUM It probably means your configuration is writing the field as a string. You can validate this my looking at the raw json and if it has quotes "1234" around the number it is a string. Logstash can set this for manipulate this for you.
Self promotion:
As alternate option might be to check out my project which parses the /proc directory to get the same statistics but I make sure they write to the Index as and integer https://github.com/eperry/logstash-input-proc Might be something your interested in
@eperry Thank your for your reply, no problem to promote your project. I will for sure have a look at it.
Regarding the Y axis; my values are correct numerical values. What if I do not want a SUM, MAX or MIN? This data is not for aggregation: I just want to have the value on a certain moment in time. On the X axis I have chosen Date Histogram and selected the timestamp field.
If your collection rate is only once an hour, and your interval is once hour. Then the aggregation only has 1 value to show MIN/Max/Average will all be the same. If your data is collected more then once an hour you need a way "summarize" the data within the interval period.
@eperry My collection has multiple observations per minute. I understand that there should be a way to summarize the buckets. But aggregating in the form of SUM, COUNT, MAX and MIN are not the ones I am looking for.
Maybe I should fetch the first or last value from the data within the interval period. I guess I need the advanced tab somehow to achieve this. Do you know how to do that? Thanks already!
Yah I have not found a way to gather the first query of an hour, It would be something definately customer/advance and that takes it out of my knowledge base.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.