MetricBeat and CloudWatch Graphs not matching up

[geek876]

Hi,
We have been using Kibana/ES since few months now and as TimeLion is now built-in in ES5 it makes a great combination. Are thinking of using Beats (metric-beats) to start with on all our servers and hence have been comparing (and learning really).

When I see MetricBeat TimeLion output v/s CloudWatch , they don't seem to match up.

As seen above,

CloudWatch: 12:46 -> Value is 7204
TimeLion : 12:46 -> Value is 13380

[andrewkroh]

Do you have more than one host or more than one interface reporting system network info to Elasticsearch? You could select one host with q=beat.name:somehost and split the graph by interfaces using s=system.network.name:5 and then see how the values compare.

[geek876]

@andrewkroh Thanks. I only have one host at the moment reporting via metricbeat and that host has only eth0. Regardless, I have now two graphs (1) Split and (2) Without split. Both rendering same values

[andrewkroh]

Can you also change CloudWatch to show 1 minute periods to match what Timelion is showing (or vice versa)? And then post the two graphs over the same exact 1 hour period.

What OS is Metricbeat running on?

[geek876]

@andrewkroh. OS is Ubuntu 15.10. Graphs Attached.

[andrewkroh]

I'm not too familiar with CloudWatch metrics, but I've been reading their docs. The default collection interval is 5 minutes, but you can increase it to 1 min to get more resolution. Metricbeat is collecting at 10 sec intervals by default. The differences could be caused by aliasing. Maybe you can try increasing the sampling interval to 1 min for the CloudWatch.

Is there a way to get the raw samples from CloudWatch in tabular form to compare to the raw samples from Metricbeat? Metricbeat is reporting the raw counter values from /proc/net/dev. CloudWatch is collecting its metrics from EC2 (not from a host agent). Since you have only one NIC I would expect the raw values to be similar (minus differences caused by sampling intervals not being aligned).

NetworkPacketsIn
The number of packets received on all network interfaces by the
instance. This metric identifies the volume of incoming traffic in
terms of the number of packets on a single instance. This metric is
available for basic monitoring only.

[geek876]

There is actually a cli to get cloudwatch logs.

aws cloudwatch get-metric-statistics --metric-name NetworkPacketsIn --namespace AWS/EC2 --statistics Maximum --dimensions Name=InstanceId,Value=xxxx --start-time 2017-02-07T11:00:00 --end-time 2017-02-07T12:00:00 --period 60

2017-02-07T11:07:00Z : 31980.0
2017-02-07T11:12:00Z : 16971.0
2017-02-07T11:17:00Z : 34274.0
2017-02-07T11:22:00Z : 46335.0
2017-02-07T11:27:00Z : 8211.0
2017-02-07T11:32:00Z : 14940.0
2017-02-07T11:37:00Z : 37422
2017-02-07T11:42:00Z : 20721
2017-02-07T11:47:00Z : 9094.0
2017-02-07T11:52:00Z : 18393.0
2017-02-07T11:57:00Z : 27123.0```

MetricBeat Graph for the same duration<img <img src="//cdck-file-uploads-us1.s3.dualstack.us-west-2.amazonaws.com/elastic/original/2X/a/aa6ee02ec2a1ddee85f87b00cd1d7a0fb5644800.png" width="690" height="226">

[ruflin]

Did you try to compare the raw values from the exact same time in the discover tab?

[geek876]

Hi,

The raw values actually do match up with Metricbeat (more closely) compared to CloudWatch.

[ruflin]

@geek876 Does that mean they are close enough to come to the conclusion, values are the same? If yes, that means metricbeat mainly gives a more detailed picture because it collects more data points?

[geek876]

@ruflin. Yes