hi. i'm trying to monitor a logstash cluster with metricbeat and logstash-xpack module. However after starting metricbeat, one datapoint is logged into elasticsearch, then an error is logged saying non-negative interval for NewTicker. I've found an old ticket for filebeat: [Filebeat] Okta Module Crashing Frequently · Issue #18530 · elastic/beats · GitHub but that seems to have been fixed, and Im not sure if applies to this anyway.
Any suggestions would be welcome.
# metricbeat version
metricbeat version 8.9.2 (amd64), libbeat 8.9.2 [d355dd57fb3accc7a2ae8113c07acb20e5b1d42a built 2023-08-30 19:42:02 +0000 UTC]
# /usr/share/logstash/bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 8.9.2
metricbeat.config.modules:
path: /etc/metricbeat/modules.d/*.yml
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
output.elasticsearch:
hosts: ["https://es:9200"]
username: "logstash-log"
password: "..."
- module: logstash
xpack.enabled: true
hosts: ["http://localhost:9600"]
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.119Z","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: metricbeat; Version: 8.9.2","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.133Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://es:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.133Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: ls","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.161Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.162Z","log.origin":{"file.name":"instance/beat.go","file.line":516},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.164Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.164Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.164Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:11.228Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":104},"message":"add_cloud_metadata: hosting provider type detected as aws, metadata={\"cloud\":{\"account\":{\"id\":\"...\"},\"availability_zone\":\"eu-west-1b\",\"image\":{\"id\":\"ami-...\"},\"instance\":{\"id\":\"i-006...\"},\"machine\":{\"type\":\"t3.medium\"},\"provider\":\"aws\",\"region\":\"eu-west-1\",\"service\":{\"name\":\"EC2\"}}}","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"error","@timestamp":"2023-09-29T09:12:11.389Z","log.origin":{"file.name":"time/tick.go","file.line":24},"message":"recovered from panic while fetching 'logstash/node_stats' for host 'localhost:9600'. Recovering, but please report this.","service.name":"metricbeat","error":{"message":"non-positive interval for NewTicker"},"stack":"github.com/elastic/elastic-agent-libs/logp.Recover\n\tgithub.com/elastic/elastic-agent-libs@v0.3.9/logp/global.go:101\nruntime.gopanic\n\truntime/panic.go:884\ntime.NewTicker\n\ttime/tick.go:24\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:228\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:209\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:149","ecs.version":"1.6.0"}
Sep 29 09:12:11 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"error","@timestamp":"2023-09-29T09:12:11.460Z","log.origin":{"file.name":"time/tick.go","file.line":24},"message":"recovered from panic while fetching 'logstash/node' for host 'localhost:9600'. Recovering, but please report this.","service.name":"metricbeat","error":{"message":"non-positive interval for NewTicker"},"stack":"github.com/elastic/elastic-agent-libs/logp.Recover\n\tgithub.com/elastic/elastic-agent-libs@v0.3.9/logp/global.go:101\nruntime.gopanic\n\truntime/panic.go:884\ntime.NewTicker\n\ttime/tick.go:24\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:228\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:209\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1\n\tgithub.com/elastic/beats/v7/metricbeat/mb/module/wrapper.go:149","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.389Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":137},"message":"Connecting to backoff(elasticsearch(https://es:9200))","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.435Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.9.2","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.445Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.9.2","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.445Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":230},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.448Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.449Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":365},"message":"Set settings.index.lifecycle.name in template to {metricbeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.463Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"metricbeat-8.9.2\" already exists and will not be overwritten.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.464Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":266},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
Sep 29 09:12:12 ls-dev-log-gergelyzs-0-i-006.... metricbeat[6918]: {"log.level":"info","@timestamp":"2023-09-29T09:12:12.466Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://es:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}