in winlogbeat.yaml I have:
hosts: ["logstashA:5044", "logstashB:5044"]
When either of those hosts is down the other one also does not get events forwarded anymore. Is this a misconfiguration on my side?
Check the logs. Does it send to both when both are up? Did it work on prior versions?
That matches our config.
During the problem, test from your winlogbeat host with telnet, maybe whatever whatever causes one to be down is really causing both to be down.
@rugenl is right, it should be sending data to the host that is up.
Make sure at least one host is up, check Winglobeat and Logstash logs to make sure they're both working correctly.
You should see log errors regarding the host that is down on Winlogbeat
I didn't test previous versions yet as this setup ran fine for 2 months.
hosts: ["logstash:5044", "localhost:5044"]
ELK@logstash is up but cannot process indices as the disk is full. With that config localhost:5044 does not show new messages anymore.
#hosts: ["logstash:5044", "localhost:5044"]
localhost:5044 immediately gets new messages.
Can I upload the logs somehow? Not sure what to look for, at least cannot see an obvious error.
Fix the disk full problem.
Yes, sure, will do so soon I was just wondering why localhost:5044 also stopped working.
It doesn't matter where the logstash servers are, if they are sending to an elastic store that is full, they can't log. They might have persistent queues and could temporarily hold some data, but diagnosing broken systems beyond the obvious break is "an exercise left to the reader" :-}
Just to make sure, disks at logstash:5044 are full, disks at localhost:5044 aren't but still localhost:5044 refuses to log / refuses to receive messages / does not get messages from Winlogbeat... I assumed that this is what a cluster is for, if one node fails others take over.