[Metricbeat - elastic module] Can't monitor more than 10 elastic nodes

aaleman · May 20, 2020, 8:55am

Hello, something weird is happens in my new cluster dockerized in swarm.

When I add 10 machine on:
`

metricbeat.modules:
- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["elasticsearch-client-01:9200", "elasticsearch-client-02:9200", "elasticsearch-master-01:9200", "elasticsearch-master-02:9200", "elasticsearch-master-03:9200", "elasticsearch-ssd-01:9200", "elasticsearch-ssd-02:9200", "elasticsearch-ssd-03:9200", "elasticsearch-ssd-04:9200", "elasticsearch-sata-01:9200"]
  username: "XXXXXX"
  password: "XXXXX" 
  xpack.enabled: true

`

Everything is right, and in kibana-monitoring tab, you can see 10 nodes monitoring with metricbeat.

The issue starts when I add 1 or n nodes more. Metricbeat monitorig max 10 hosts and rest is monitoring by itself. Changing every X period of time node that metricbeat is monitoring.

hosts: ["elasticsearch-client-01:9200", "elasticsearch-client-02:9200", "elasticsearch-master-01:9200", "elasticsearch-master-02:9200", "elasticsearch-master-03:9200", "elasticsearch-ssd-01:9200", "elasticsearch-ssd-02:9200", "elasticsearch-ssd-03:9200", "elasticsearch-ssd-04:9200", "elasticsearch-sata-01:9200", "elasticsearch-sata-02:9200", "elasticsearch-sata-03:9200", "elasticsearch-sata-04:9200"]

Do you know if exist some limitation like redis module (maxcon) ? Do you know how to avoid this limitation and monitor more than 10 hosts in elasticsearch module?

aaleman · May 20, 2020, 8:56am

I don't have any log about this

|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:51:55.855Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1510220,"time":{"ms":964}},"total":{"ticks":3596020,"time":{"ms":2609},"value":3596020},"user":{"ticks":2085800,"time":{"ms":1645}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":59940129}},"memstats":{"gc_next":19700272,"memory_alloc":11660968,"memory_total":190739877608},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":109,"batches":21,"duplicates":96,"total":205}},"pipeline":{"clients":120,"events":{"active":2,"published":207,"total":207},"queue":{"acked":205}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.29,"15":0.2,"5":0.25,"norm":{"1":0.0483,"15":0.0333,"5":0.0417}}}}}}|
|---|---|---|---|---|---|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:52:25.856Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1510840,"time":{"ms":623}},"total":{"ticks":3597780,"time":{"ms":1772},"value":3597780},"user":{"ticks":2086940,"time":{"ms":1149}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":59970128}},"memstats":{"gc_next":30927680,"memory_alloc":27164216,"memory_total":190836436344},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":113,"batches":22,"duplicates":96,"total":209}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":209}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.23,"15":0.2,"5":0.24,"norm":{"1":0.0383,"15":0.0333,"5":0.04}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:52:55.855Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1511810,"time":{"ms":965}},"total":{"ticks":3600130,"time":{"ms":2336},"value":3600130},"user":{"ticks":2088320,"time":{"ms":1371}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60000131}},"memstats":{"gc_next":19218400,"memory_alloc":15983216,"memory_total":190932888064},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":21,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.14,"15":0.19,"5":0.22,"norm":{"1":0.0233,"15":0.0317,"5":0.0367}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:53:25.856Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1512610,"time":{"ms":801}},"total":{"ticks":3602320,"time":{"ms":2198},"value":3602320},"user":{"ticks":2089710,"time":{"ms":1397}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60030128}},"memstats":{"gc_next":19477568,"memory_alloc":11677472,"memory_total":191029465488},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":22,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.08,"15":0.18,"5":0.2,"norm":{"1":0.0133,"15":0.03,"5":0.0333}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:53:55.856Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1513340,"time":{"ms":732}},"total":{"ticks":3604260,"time":{"ms":1942},"value":3604260},"user":{"ticks":2090920,"time":{"ms":1210}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60060130}},"memstats":{"gc_next":30672496,"memory_alloc":27275952,"memory_total":191126162280},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":21,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.17,"15":0.19,"5":0.21,"norm":{"1":0.0283,"15":0.0317,"5":0.035}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:54:25.856Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1514300,"time":{"ms":964}},"total":{"ticks":3606970,"time":{"ms":2706},"value":3606970},"user":{"ticks":2092670,"time":{"ms":1742}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60090128}},"memstats":{"gc_next":19748304,"memory_alloc":13331784,"memory_total":191222402368},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":21,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.1,"15":0.18,"5":0.19,"norm":{"1":0.0167,"15":0.03,"5":0.0317}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:54:55.855Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1514990,"time":{"ms":692}},"total":{"ticks":3608980,"time":{"ms":2012},"value":3608980},"user":{"ticks":2093990,"time":{"ms":1320}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60120129}},"memstats":{"gc_next":19726976,"memory_alloc":13509672,"memory_total":191318817632},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":21,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.06,"15":0.17,"5":0.17,"norm":{"1":0.01,"15":0.0283,"5":0.0283}}}}}}|
|optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T08:55:25.856Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1515610,"time":{"ms":612}},"total":{"ticks":3610750,"time":{"ms":1764},"value":3610750},"user":{"ticks":2095140,"time":{"ms":1152}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0503f3c-ea57-49e4-9c2b-1fb3a07f3b2e","uptime":{"ms":60150128}},"memstats":{"gc_next":36261824,"memory_alloc":27043176,"memory_total":191415177448},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":111,"batches":21,"duplicates":96,"total":207}},"pipeline":{"clients":120,"events":{"active":0,"published":207,"total":207},"queue":{"acked":207}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":45,"success":45},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":96,"success":96}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.1,"15":0.17,"5":0.17,"norm":{"1":0.0167,"15":0.0283,"5":0.0283}}}}}}|

mtojek · May 20, 2020, 10:17am

I don't know nothing about any limitations. Can you try enabling the debug mode and see what's going on deeper?

aaleman · May 20, 2020, 12:10pm

Thanks for your reply, for now, all logs from start and nothing relevant in info: I will try with debug but is insane, a lot of lines...wish me lucky

optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T11:49:36.261Z	INFO	[api]	api/server.go:66	Stats endpoint (127.0.0.1:5066) finished: accept tcp 127.0.0.1:5066: use of closed network connection
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T11:57:40.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":16910,"time":{"ms":850}},"total":{"ticks":35930,"time":{"ms":2085},"value":35930},"user":{"ticks":19020,"time":{"ms":1235}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":480145}},"memstats":{"gc_next":19428288,"memory_alloc":12375032,"memory_total":1650034128},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.15,"15":0.11,"5":0.11,"norm":{"1":0.025,"15":0.0183,"5":0.0183}}}}}}
optoslgcy-dev_metricbeat.1.y94mq6m2gf5e@dev-worker12    | 2020-05-20T11:49:36.290Z	INFO	instance/beat.go:444	metricbeat stopped.
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T11:58:10.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":18440,"time":{"ms":1533}},"total":{"ticks":38890,"time":{"ms":2967},"value":38890},"user":{"ticks":20450,"time":{"ms":1434}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":510148}},"memstats":{"gc_next":19296224,"memory_alloc":16083328,"memory_total":1751442504},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.14,"15":0.11,"5":0.12,"norm":{"1":0.0233,"15":0.0183,"5":0.02}}}}}}
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T11:58:40.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":19740,"time":{"ms":1297}},"total":{"ticks":41180,"time":{"ms":2286},"value":41180},"user":{"ticks":21440,"time":{"ms":989}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":540148}},"memstats":{"gc_next":19542848,"memory_alloc":12284992,"memory_total":1853111760},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.16,"15":0.11,"5":0.12,"norm":{"1":0.0267,"15":0.0183,"5":0.02}}}}}}
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T11:59:10.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":21120,"time":{"ms":1375}},"total":{"ticks":43990,"time":{"ms":2809},"value":43990},"user":{"ticks":22870,"time":{"ms":1434}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":570146}},"memstats":{"gc_next":19314208,"memory_alloc":15574032,"memory_total":1954682688},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.1,"15":0.11,"5":0.11,"norm":{"1":0.0167,"15":0.0183,"5":0.0183}}}}}}
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T11:59:40.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":21970,"time":{"ms":857}},"total":{"ticks":46320,"time":{"ms":2334},"value":46320},"user":{"ticks":24350,"time":{"ms":1477}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":600146}},"memstats":{"gc_next":19444336,"memory_alloc":10934336,"memory_total":2056204768},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.17,"15":0.11,"5":0.13,"norm":{"1":0.0283,"15":0.0183,"5":0.0217}}}}}}
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T12:00:10.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":22900,"time":{"ms":925}},"total":{"ticks":48570,"time":{"ms":2240},"value":48570},"user":{"ticks":25670,"time":{"ms":1315}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":630147}},"memstats":{"gc_next":19387792,"memory_alloc":13232632,"memory_total":2157878232},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":0.16,"15":0.11,"5":0.13,"norm":{"1":0.0267,"15":0.0183,"5":0.0217}}}}}}
optoslgcy-dev_metricbeat.1.m5zdmk1xf2ws@dev-worker12    | 2020-05-20T12:00:40.973Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":24960,"time":{"ms":2058}},"total":{"ticks":51510,"time":{"ms":2945},"value":51510},"user":{"ticks":26550,"time":{"ms":887}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":153},"info":{"ephemeral_id":"f0506a3f-c756-4d26-8015-2a15aaa66295","uptime":{"ms":660145}},"memstats":{"gc_next":19240192,"memory_alloc":15628312,"memory_total":2259282272},"runtime":{"goroutines":790}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":114,"batches":18,"duplicates":102,"total":216}},"pipeline":{"clients":120,"events":{"active":0,"published":216,"total":216},"queue":{"acked":216}}},"metricbeat":{"beat":{"state":{"events":3,"success":3},"stats":{"events":3,"success":3}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":6,"success":6},"index":{"events":48,"success":48},"index_recovery":{"events":3,"success":3},"index_summary":{"events":3,"success":3},"node_stats":{"events":39,"success":39},"shard":{"events":102,"success":102}},"kibana":{"stats":{"events":6,"success":6}}},"system":{"load":{"1":1.25,"15":0.2,"5":0.39,"norm":{"1":0.2083,"15":0.0333,"5":0.065}}}}}}

system · June 17, 2020, 2:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.