Metricbeat dashboards doesnot show all data

[Gauti (Gautham)]

Hey Guys,

I have installed metricbeat in some virtual machines and started sending data to elasticsearch. I have imported default dashboards also for the same.

The problem is, the dashboard is not visualizing all data, instead its showing only for 3 VM's. If i convert the timing to last 7 days its visualizing, but for 15 or 30 mins time category i'm not getting all the data.

Screeenshot FYI...

Screenshot for 7days

NOTE : The data is available when i check in discover tab.

Any suggestions?

Thanks
Gautham

[ruflin]

Could you check the timestamps of the data points if they are all as they should be or if there are some offsets? What is the minimum duration to select where all data shows up?

[Gauti (Gautham)]

@ruflin these servers are from different locations, so the time differs from server to server.

I discover tab i'm able to see the enter is made as per elasticsearch server timing.

Minumum i had set to 15minutes in the dashboard.

Thanks
Gautham

[ruflin]

Which field did you look at for the timestamps?

If I understand it right, with 15 minutes you don't see all the data. What is the minimum you see all data ? For example 3 hours or 24 hours? Or only 7 days?

[Gauti (Gautham)]

@ruflin when i reduce dthe time to 15mins i get only 2 servers, sometimes 1 server.
When i use 24 hours or 7 days i'm able to see all the hosts monitored.

Thanks
Gauti

[ruflin]

That kind of indicates for me that there is something up with the timestamps / time difference.

I assume you always see the same 1-2 servers in the last 15 minutes? Could you share 1 event for each of this server and 1-2 events of servers that you only see when you set it to 24h?

[Gauti (Gautham)]

HI @ruflin yeah u are ryt i always see 1 or 2 servers in last 15mins.......here is the event data.

Event for last 30mins:
@timestamp:June 21st 2018, 18:39:41.083 metricset.module:system metricset.rtt:20,001 metricset.name:process system.process.memory.size:10.695MB system.process.memory.rss.pct:0.22% system.process.memory.rss.bytes:18.418MB system.process.memory.share:0B system.process.ppid:500 system.process.pgid:0 system.process.username:NT AUTHORITY\SYSTEM system.process.pid:616 system.process.name:lsass.exe system.process.state:running

Event for last 7 days:
June 21st 2018, 18:19:34.328 beat.hostname:ITSM @timestamp:June 21st 2018, 18:19:34.328 metricset.rtt:36,132 metricset.name:process metricset.module:system system.process.state:running system.process.pid:3,240 system.process.memory.size:3.9GB system.process.memory.rss.bytes:2.159GB system.process.memory.rss.pct:26.99% system.process.memory.share:0B system.process.cmdline:"D:\BMCSoftware\ARSystem\arserver.exe" --unicode -i "D:\BMCSoftware\ARSystem" -l "C:\Program Files\Common Files\AR System\Licenses\onbmc-s" -m system.process.ppid:3,172 system.process.cpu.total.value:775,883,375 system.process.cpu.total.pct:103.44% system.process.cpu.total.norm.pct:25.86% system.process.cpu.start_time:June 13th 2018, 00:10:28.197 system.process.name:arserver.exe system.process.pgid:0 system.process.username:NT AUTHORITY\SYSTEM beat.name:ITSM beat.version:6.2.4 _id:00pkImQBUOoQn_4AqYAl _type:doc _index:metricbeat-6.2.4-2018.06.21 _score: -

June 21st 2018, 18:20:38.556 beat.hostname:BMCSQL @timestamp:June 21st 2018, 18:20:38.556 metricset.name:memory metricset.module:system system.memory.used.bytes:10.982GB system.memory.used.pct:91.52% system.memory.free:1.018GB system.memory.actual.free:1.018GB system.memory.actual.used.bytes:10.982GB system.memory.actual.used.pct:91.52% system.memory.swap.total:23.997GB system.memory.swap.used.bytes:12.196GB

Thanks
Gautham

[ruflin]

So the event you shared under 7 days does not show up in the "last 30 minutes" when you searched for it? But later it shows up with the exact timestamp?

I would like to figure out what it is the minimal period you have to set to see all events. It seems not only with 7 days you see everything but also with 24 hours? What about 1h, 2h ?

Are all your VM's in the same time zone? Do they have different setup?

[Gauti (Gautham)]

@ruflin Yeah you are rite it shows up with exact time stamp only wen i choose 7 days or more. I'm not getting any details even if i choose 24hr or 1hr or 2hr.

For all these hours 24hr or 1hr or 2hr : I'm getting only one VM.

If i choose 7 days or more : I'll get to see all other vm's.

Few Vm's are running on different time as well.

As a best practice i would like to set a minimal of 15 or 30mins, so that the NOC team can keep watching the server performance.

Thanks
Gautham

[ruflin]

The part I'm struggling is that you mentioned in your first post that the data shows up in discovery.

What do you get when you click on a single host that is not shown when you only look at the last 5 minutes and then scroll down to see cpu usage for example? Do you see the CPU usage for all 7 days or is the last part of the visualisation empty? See here as an example.

[Gauti (Gautham)]

hey @ruflin when i choose single host it doesnot have any information, like all the metrics shows 0% except the process count.

Due to some other issue i ended up rebooting my clutser last night, post that i was able to get the data in dashboard properly,

here is the screenshot for 15mins data

Was just wondering in case the same issue arise again, where should i need to look or should i need to restart cluster everytime when such issue happens.

Thanks
Gautham

[ruflin]

I really have a hard time to understand on what happened here. A cluster restart should definitively not be needed. As after your cluster restart it seems all the data showed up, not only since the restart so all the data was there all the time as you described in the beginning.

I'm happy it works now, in case it breaks again, lets investigate further.