Regression? Metricbeat dies immediately if kibana isn't running yet (or is still starting)

[archon810 (Artem Russakovskii)]

This used to work in v7, but broken in v8, which we upgraded to recently.

Restarting kibana and then restarting metricbeat, like so:

systemctl restart elasticsearch && systemctl restart kibana && systemctl restart metricbeat && systemctl restart auditbeat && systemctl restart filebeat

results in metricbeat dying with the following right away, without waiting any number of seconds to retry:

{"log.level":"error","@timestamp":"2023-03-28T14:34:32.944-0700","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: 4 errors: error making http request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused; error making http request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused; error making http request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused; error making http request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}

Note that neither filebeat nor auditbeat suffer from this issue. This feels like a regression bug, because it never used to be a problem in v7. And it's not like kibana is not there, it's there, but still going through its starting paces.

There are already max_retries and related retry settings for elasticsearch retries, there should probably be a similar set of (working) configs for kibana?

Here's a relevant ticket on Github Metricbeat fails to start if kibana module is down · Issue #34135 · elastic/beats · GitHub. Will someone be able to give it some attention please?

[Wave (Andrew Sullivan)]

Hi @archon810,

So my understanding of the command you provide is each command will run from left to right assuming the left most command completes successfully. First elasticsearch, then kibana, however once kibana restarts metricbeat is still running because it hasn't restarted yet. Could you do the following pseudocode steps?

1. Stop metricbeat
2. Stop auditbeat
3. Stop filebeat
4. Stop kibana
5. Restart elastic
6. Start kibana
7. Start filebeat
8. Start auditbeat
9. Start metricbeat

[archon810 (Artem Russakovskii)]

Hi Andrew,

I've reconfigured the order, but it still predictably does the same thing, errors out, and quits.

systemctl stop metricbeat && systemctl stop auditbeat && systemctl stop filebeat && systemctl stop kibana && systemctl restart elasticsearch && systemctl start kibana && systemctl start filebeat && systemctl start auditbeat && systemctl start metricbeat

In the meantime, I've upgraded everything to 8.7, and the same issue persists.

[archon810 (Artem Russakovskii)]

The bug was identified and should hopefully be fixed in a future release: Metricbeat fails to start if kibana module is down · Issue #34135 · elastic/beats · GitHub.