Visualising rate of change of a field

Elastic Stack Kibana
1

I'm just getting started with ELK, and we're wanting to use Kibana to visualise the performance of some of our servers.

In the first instance, I'm trying to put some network usage data in using collectd on a few servers. Collectd is sending through a record with a type of if_octets containing fields called rx and tx. These fields contain a count of the number of octets received and sent through the network interface. As great as this is, I'd like to visualise these in terms of the amount they change (ie I'd like to see rx octets per second and tx octets per second). I have similar requirements for the various records collectd can emit from its cpu plugin.

Searching for an easy way to do this is coming up blank for me. The only way I've found so far is to use the elasticsearch filter in logstash to pull in the value and timestamp from the previous record and put those through a ruby filter to calculate the metric I want. This works, but my small test environment blocked pretty quickly once I started looking up records in the filter for more than just a couple of the types that collectd is sending through. The other downside is that I'm having to calculate this when the record is initially processed by logstash. If I identify another similar field that I need to have this sort of visualisation for at some point in the future, I'll have to find a way to modify the already indexed documents in elasticsearch to add this, or miss out on having it available for historical data.

So, does anyone know of a better way to do this sort of thing?

Thanks

2

At the moment, there isn't a better way to accomplish adding a derivative in Kibana 4, other than indexing the value outside of Kibana (as you are doing).

However, Elasticsearch 2.0 has added a derivative aggregation on the backend, which Kibana can utilize in the future for a way to do this directly in the visualization. Feel free to +1 the issue! https://github.com/elastic/kibana/issues/1743

3

@martin_goldstone, i have the same problem with collectd and some metrcis usage.

Did you manage to get around this?

Thanks

4

Just an update,

I have not "fixed" the kibana thing. but at least for CPU with collectd as version 5.5 you can report as percent
So put this option at the plugin cpu

    ValuesPercentage True

Hope that helps. Sorry for the code format

5

Hi all,
Did you all manage to figure out the procedure for network throughput ?

Thank you

6

Havent checked that yet.

Only CPU was the problem at the moment.

Do you know any other plugins with problem?

7

Hi,
Yes with the interface plugin. I was interested in getting the results with bytes/sec ? but have no idea, how this can be achieved with interface plugin?

<Node "example">
Host "localhost"
Port "2003"
Protocol "tcp"
StoreRates true

8

Try to change the datasource, dont know if it supports it, but is worth a try.
look here for "datasource"
https://collectd.org/documentation/manpages/collectd.conf.5.shtml

9

+1 for interface and disk plugins and any plugin that return a counter value rather that the derivative of value, maybe we could get Kibana to utilize 'Elasticsearch 2.0 has added a derivative aggregation on the backend'?