Fleet Agents hitting 100% CPU usage

Elastic Stack Beats
,
1

We put the elastic-agent for fleet onto our VDI infrastructure. Each VDI machine started out with 2 vCPUs and 4 GB RAM. We installed the elastic-agent through fleet. The machine performed fine for a few hours then became unusable. Seen CPU and RAM usage spikes. Raised each machine with the elastic-agent installed to 4 vCPUs and 16 GB RAM. Still seeing the vdi machines hit in the high 80's and 90's percentile usage of CPU and RAM. Is there a known issue with this?

I am also seeing spikes on regular workstations (laptops) that run the elastic agent, the difference is that the agent will ramp up a high CPU usage for a minute or two then settle back down to normal usage. Does this two to three times a day.

MicrosoftTeams-image
MicrosoftTeams-image1595×387 56.9 KB

2

Could you share the exact version of Elastic Agent you are using? Also any chance to share some logs around the time it peaks?

3

I am sorry, for sounding like a newbie but can you let me know the location as to what logs I should get?

4

I am sorry, I also forgot the version we are currently installing version 7.11.2 of the agent. Once it is installed I upgrade them to 7.12.

5

It is a bit (too) hard to find the logs at the moment. It is why I recently filed [elastic-agent] logs and metrics command · Issue #25375 · elastic/beats · GitHub You can find the logs in the data/{your-agent-id}/logs directory. There should be one for the elastic-agent and each process it runs.

6

And what I forgot, give some indications of the policy you use. If you go to Kibana, you can show it as yaml.

7

@ruflin I have the logs now, but there are a lot of logs. Is there a particular log that you want to take a look at? I currently have the following logs:

elastic-agent-json.log
elastic-agent-watcher-json.log

Then in the "default" folder there are the following logs:

filebeat_monitor-json.log
filebeat-json.log
metricbeat_monitor-json.log
metricbeat-json.log

As for what is being applied in the policy is "System" and "Windows" which is the logs.

8

Here is the logs from each of the previously listed logs

elastic-agent-json.log

{"log.level":"error","@timestamp":"2021-04-28T16:05:56.168-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.173-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-04-28T16:05:56.173-0400","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-04-28T16:05:56-04:00: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.174-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":176},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf ERROR {13842501837555687300 55481828268001 0x29f38a0} FAILED Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.174-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:56-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.175-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532436718408652 83977208413401 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.175-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:56-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.175-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532436719408852 83977209413501 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:58.024-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:58-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:58.024-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532438715896800 83979058273701 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:58.525-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:58-04:00: type: 'STATE': sub_type: 'CONFIG' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to CONFIG: Updating configuration","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:58.526-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532439216938000 83979559276001 0x29f38a0} CONFIG Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to CONFIG: Updating configuration map[]}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-04-28T16:05:59.026-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-04-28T16:05:59.026-0400","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-04-28T16:05:59-04:00: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.027-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":176},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf ERROR {13842501840524417372 55484575514601 0x29f38a0} FAILED Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.031-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.031-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:59-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.032-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532439796721324 83980065278101 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.032-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:05:59-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:59.032-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532439797721124 83980066277801 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:00.876-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:06:00-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:00.877-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532441714993648 83981909665401 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.377-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:06:01-04:00: type: 'STATE': sub_type: 'CONFIG' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to CONFIG: Updating configuration","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.378-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532442289732572 83982410623701 0x29f38a0} CONFIG Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to CONFIG: Updating configuration map[]}","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-04-28T16:06:01.878-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-04-28T16:06:01.878-0400","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-04-28T16:06:01-04:00: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.879-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":176},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf ERROR {13842501843508336144 55487337950001 0x29f38a0} FAILED Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to FAILED: 1 error: 1 error: Error creating runner from config: 1 error: metricset 'system/load' not found map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.883-0400","log.origin":{"file.name":"status/reporter.go","file.line":233},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.886-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:06:01-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.886-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532442797771572 83982918623401 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.886-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:06:01-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:01.887-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532442798771272 83982919623001 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to RESTARTING: Restarting map[]}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:03.688-0400","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-04-28T16:06:03-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:03.689-0400","log.origin":{"file.name":"fleet/reporter.go","file.line":162},"message":"fleet reporter dropped event because threshold[10000] was reached: &{e1737370-a390-11eb-9d7c-4356dcebfacf STATE {13842532444748390620 83984721619301 0x29f38a0} STARTING Application: metricbeat--7.12.0[e1737370-a390-11eb-9d7c-4356dcebfacf]: State changed to STARTING: Starting map[]}","ecs.version":"1.6.0"}
9

Continued ...

elastic-agent-watcher-json.log

{"log.level":"error","@timestamp":"2021-04-27T16:46:12.718-0400","log.origin":{"file.name":"cmd/watch.go","file.line":61},"message":"failed to load markeropen C:\\Program Files\\Elastic\\Agent\\data\\.update-marker: The system cannot find the file specified.","ecs.version":"1.6.0"}

filebeat_monitor-json.log

{"log.level":"info","@timestamp":"2021-04-28T16:00:04.166-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9937,"value":9937},"user":{"ticks":8078}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83580377}},"memstats":{"gc_next":19659936,"memory_alloc":9836184,"memory_total":291620272,"rss":50741248},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:00:33.990-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9937,"value":9937},"user":{"ticks":8078}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83610197}},"memstats":{"gc_next":19659936,"memory_alloc":9951712,"memory_total":291735800,"rss":50761728},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:01:03.993-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9952,"time":{"ms":15},"value":9952},"user":{"ticks":8093,"time":{"ms":15}}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83640200}},"memstats":{"gc_next":19659936,"memory_alloc":10016128,"memory_total":291800216,"rss":50761728},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:01:33.993-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9952,"value":9952},"user":{"ticks":8093}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83670197}},"memstats":{"gc_next":19659936,"memory_alloc":10093280,"memory_total":291877368,"rss":50761728},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:02:03.997-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"time":{"ms":16},"value":9968},"user":{"ticks":8109,"time":{"ms":16}}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83700197}},"memstats":{"gc_next":19665376,"memory_alloc":9839448,"memory_total":291945248,"rss":50761728},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:02:34.013-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83730200}},"memstats":{"gc_next":19665376,"memory_alloc":9910984,"memory_total":292016784,"rss":50761728},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:03:04.041-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83760237}},"memstats":{"gc_next":19665376,"memory_alloc":9972568,"memory_total":292078368,"rss":50769920},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:03:34.014-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":259},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83790198}},"memstats":{"gc_next":19665376,"memory_alloc":10045320,"memory_total":292151120,"rss":50769920},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:04:04.008-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83820197}},"memstats":{"gc_next":19661632,"memory_alloc":9836952,"memory_total":292217416,"rss":50757632},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:04:34.012-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83850197}},"memstats":{"gc_next":19661632,"memory_alloc":9907064,"memory_total":292287528,"rss":50757632},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:04.013-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83880197}},"memstats":{"gc_next":19661632,"memory_alloc":9977528,"memory_total":292357992,"rss":50774016},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:34.017-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1859},"total":{"ticks":9968,"value":9968},"user":{"ticks":8109}},"handles":{"open":257},"info":{"ephemeral_id":"92b86b6d-1d22-4b37-8392-b10a6ccf5a0e","uptime":{"ms":83910197}},"memstats":{"gc_next":19661632,"memory_alloc":10047288,"memory_total":292427752,"rss":50774016},"runtime":{"goroutines":37}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
10

filbeat-json.log

{"log.level":"info","@timestamp":"2021-04-28T16:00:03.931-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59562,"time":{"ms":16}},"total":{"ticks":103327,"time":{"ms":31},"value":103327},"user":{"ticks":43765,"time":{"ms":15}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83580440}},"memstats":{"gc_next":20746432,"memory_alloc":11041696,"memory_total":1934241408,"rss":55046144},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:00:33.793-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59562},"total":{"ticks":103343,"time":{"ms":16},"value":103343},"user":{"ticks":43781,"time":{"ms":16}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83610419}},"memstats":{"gc_next":20746432,"memory_alloc":11743504,"memory_total":1934943216,"rss":55074816},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:01:03.793-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59578,"time":{"ms":16}},"total":{"ticks":103359,"time":{"ms":16},"value":103359},"user":{"ticks":43781}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83640419}},"memstats":{"gc_next":20746432,"memory_alloc":12400648,"memory_total":1935600360,"rss":55074816},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:01:33.795-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59609,"time":{"ms":31}},"total":{"ticks":103390,"time":{"ms":31},"value":103390},"user":{"ticks":43781}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83670419}},"memstats":{"gc_next":20749696,"memory_alloc":10377280,"memory_total":1936268528,"rss":55087104},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:02:03.801-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59640,"time":{"ms":31}},"total":{"ticks":103421,"time":{"ms":31},"value":103421},"user":{"ticks":43781}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83700420}},"memstats":{"gc_next":20749696,"memory_alloc":11037920,"memory_total":1936929168,"rss":55087104},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:02:33.894-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59640},"total":{"ticks":103421,"value":103421},"user":{"ticks":43781}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83730456}},"memstats":{"gc_next":20749696,"memory_alloc":11697928,"memory_total":1937589176,"rss":55091200},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:03:03.814-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59656,"time":{"ms":16}},"total":{"ticks":103452,"time":{"ms":31},"value":103452},"user":{"ticks":43796,"time":{"ms":15}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83760421}},"memstats":{"gc_next":20749696,"memory_alloc":12355984,"memory_total":1938247232,"rss":55091200},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:03:33.810-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59671,"time":{"ms":15}},"total":{"ticks":103483,"time":{"ms":31},"value":103483},"user":{"ticks":43812,"time":{"ms":16}}},"handles":{"open":288},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83790421}},"memstats":{"gc_next":20732416,"memory_alloc":10379592,"memory_total":1938923824,"rss":55091200},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:04:03.811-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59687,"time":{"ms":16}},"total":{"ticks":103562,"time":{"ms":79},"value":103562},"user":{"ticks":43875,"time":{"ms":63}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83820419}},"memstats":{"gc_next":20732416,"memory_alloc":11034240,"memory_total":1939578472,"rss":55074816},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:04:33.814-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59734,"time":{"ms":47}},"total":{"ticks":103655,"time":{"ms":93},"value":103655},"user":{"ticks":43921,"time":{"ms":46}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83850419}},"memstats":{"gc_next":20732416,"memory_alloc":11692040,"memory_total":1940236272,"rss":55074816},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:03.817-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59750,"time":{"ms":16}},"total":{"ticks":103718,"time":{"ms":63},"value":103718},"user":{"ticks":43968,"time":{"ms":47}}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83880420}},"memstats":{"gc_next":20732416,"memory_alloc":12369600,"memory_total":1940913832,"rss":55091200},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:33.819-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":59781,"time":{"ms":31}},"total":{"ticks":103749,"time":{"ms":31},"value":103749},"user":{"ticks":43968}},"handles":{"open":286},"info":{"ephemeral_id":"d1934cdb-62dd-49a4-a858-163d0a688297","uptime":{"ms":83910419}},"memstats":{"gc_next":20731712,"memory_alloc":10362608,"memory_total":1941558592,"rss":55091200},"runtime":{"goroutines":40}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
11

metricbeat_monitor-json.log

{"log.level":"info","@timestamp":"2021-04-28T16:04:56.819-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:04:56.819-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:04:56.820-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:05.965-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":10703},"total":{"ticks":54390,"value":54390},"user":{"ticks":43687}},"handles":{"open":298},"info":{"ephemeral_id":"0c1d335b-825a-4bc6-9318-9a016234f042","uptime":{"ms":83881482}},"memstats":{"gc_next":19202928,"memory_alloc":10890936,"memory_total":4001176864,"rss":65486848},"runtime":{"goroutines":87}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":30,"active":80,"batches":1,"total":20},"read":{"bytes":903},"write":{"bytes":30593}},"pipeline":{"clients":10,"events":{"active":50,"published":30,"total":30},"queue":{"acked":30}}},"metricbeat":{"beat":{"state":{"events":9,"failures":4,"success":5},"stats":{"events":9,"failures":4,"success":5}},"http":{"json":{"events":12,"failures":4,"success":8}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:06.819-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:06.819-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:06.820-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:16.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:16.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:16.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.821-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.821-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.822-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:26.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.026-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Non-zero metrics in the last 30s","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":10703},"total":{"ticks":54406,"time":{"ms":16},"value":54406},"user":{"ticks":43703,"time":{"ms":16}}},"handles":{"open":298},"info":{"ephemeral_id":"0c1d335b-825a-4bc6-9318-9a016234f042","uptime":{"ms":83911542}},"memstats":{"gc_next":19202928,"memory_alloc":12262672,"memory_total":4002548600,"rss":65495040},"runtime":{"goroutines":87}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":20,"active":90,"batches":1,"total":30},"read":{"bytes":808},"write":{"bytes":45367}},"pipeline":{"clients":10,"events":{"active":60,"published":30,"total":30},"queue":{"acked":20}}},"metricbeat":{"beat":{"state":{"events":9,"failures":4,"success":5},"stats":{"events":9,"failures":4,"success":5}},"http":{"json":{"events":12,"failures":4,"success":8}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.822-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.822-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:46.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:46.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:46.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.823-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.824-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.825-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.825-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:05:56.825-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\default-metricbeat: The system cannot find the file specified.","ecs.version":"1.6.0"}
12

metricbeat-json.log

{"log.level":"info","@timestamp":"2021-04-28T16:06:02.757-0400","log.origin":{"file.name":"instance/beat.go","file.line":660},"message":"Home path: [C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64] Config path: [C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64] Data path: [C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\run\\default\\metricbeat--7.12.0] Logs path: [C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64\\logs]","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.762-0400","log.origin":{"file.name":"instance/beat.go","file.line":668},"message":"Beat ID: 149239ca-f9ef-4b7c-a4e1-5977420e99be","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.811-0400","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.811-0400","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: \\\\.\\pipe\\default-metricbeat (configured: npipe:///default-metricbeat)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.811-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":996},"message":"Beat info","system_info":{"beat":{"path":{"config":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64","data":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\run\\default\\metricbeat--7.12.0","home":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64","logs":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64\\logs"},"type":"metricbeat","uuid":"149239ca-f9ef-4b7c-a4e1-5977420e99be"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.811-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1005},"message":"Build info","system_info":{"build":{"commit":"08e20483a651ea5ad60115f68ff0e53e6360573a","libbeat":"7.12.0","time":"2021-03-18T06:28:33.000Z","version":"7.12.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.811-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1008},"message":"Go runtime info","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.15.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.817-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":105},"message":"add_cloud_metadata: hosting provider type detected as azure, metadata={\"account\":{},\"instance\":{\"id\":\"3d5e44d8-b4c4-40fd-ba5a-c04dcdc94fd6\",\"name\":\"WVDDBAPR01\"},\"machine\":{\"type\":\"Standard_D4s_v3\"},\"provider\":\"azure\",\"region\":\"eastus\"}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.826-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1012},"message":"Host info","system_info":{"host":{"architecture":"x86_64","boot_time":"2021-04-27T16:45:32.1-04:00","name":"WVDDBAPR01","ip":["fe80::f87e:1943:91ad:f62a/64","172.25.101.4/27","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.1500 (WinBuild.160101.0800)","mac":["00:0d:3a:12:ae:9a"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise for Virtual Desktops","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.1500"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"7a2edb5b-bdab-4eb3-9e0d-47bcf07b6935"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.826-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1041},"message":"Process info","system_info":{"process":{"cwd":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64","exe":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-08e204\\install\\metricbeat-7.12.0-windows-x86_64\\metricbeat.exe","name":"metricbeat.exe","pid":1016,"ppid":3816,"start_time":"2021-04-28T16:06:01.885-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.826-0400","log.origin":{"file.name":"instance/beat.go","file.line":304},"message":"Setup Beat: metricbeat; Version: 7.12.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.827-0400","log.origin":{"file.name":"eslegclient/connection.go","file.line":99},"message":"elasticsearch url: http://localhost:9200","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.827-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: WVDDBAPR01","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.868-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":117},"message":"Starting metrics logging every 30s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.868-0400","log.origin":{"file.name":"instance/beat.go","file.line":468},"message":"metricbeat start running.","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-04-28T16:06:02.868-0400","log.logger":"cfgwarn","log.origin":{"file.name":"fleet/manager.go","file.line":101},"message":"BETA: Fleet management is enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.868-0400","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"fleet/manager.go","file.line":102},"message":"Starting fleet management service","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.869-0400","log.origin":{"file.name":"service/service_windows.go","file.line":122},"message":"Attempted to register Windows service handlers, but this is not a service. No action necessary","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.869-0400","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-04-28T16:06:02.870-0400","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","ecs.version":"1.6.0"}

Thanks,

13

Few things from the logs:

  • system/load is not available on the windows machine. Could you disable it in the UI? (should not cause the spike)
  • Something around the self monitoring of the agent is not working. This might be windows related:
{"log.level":"info","@timestamp":"2021-04-28T16:05:36.822-0400","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://npipe/state\": open \\\\.\\pipe\\default-endpoint-security: The system cannot find the file specified.","ecs.version":"1.6.0"}

@michalp Do you know more here? ^

Any chance you could try to disable the monitor of the Elastic Agent for this policy and see if you still see the spikes?

14

@ruflin,

I toggled the system load metrics switch in the UI to off. As for the agent's self monitoring I am not sure as to how to turn that off. I know that there is an integration for "Elastic-Agent", I thought that is what is monitoring the performance and stuff of the agent. I do not have that added into the policy. As I stated before, I am only using the "System" and "Windows" integrations on the VDI Policy.

15

I found a setting in the policy to disable the agent monitoring. I have disabled that feature as of now. I will let it run today, and collect more logs to see if that changes anything.

Thanks,

16

Great thanks!

17

So it seems that by disabling the agent self monitoring feature the CPU and RAM have settled back down to 2% usage.

Not sure if that was the cause of the bug or not but if you would like to to keep testing other items we can.

Thanks,

18

Thanks for digging into this. We made quite a few changes for 7.13 and I'm now wondering if assuming this is a but, if it is still around. But 7.13 is not out yet at the same time it means we would still have time to fix it. @steffens Any ideas for the above on what could cause it? Any additional logs / metrics that would help?

19

@ruflin
I had may be same issue 7.12.0 running on test box was very very slow and found this hard way.
by default it enabled fleet update and it slow down everything.

I had to put this in kibana to bring everything back to normal. My test system is not connected to internet and hence the trouble

xpack.fleet.enabled: false

20

Thanks for the additional details @elasticforme Definitively something we should test on how things behave in a non connected environment.