From time to time most of my configured HTTP monitors fail with next errors:
read: connection reset by peer
or
http: request timed out while waiting for response (Client.Timeout exceeded while awaiting headers)
or
EOF
I tried to increase schedule interval for them form 1 minute to 2 or 3, it a bit helped since I started less frequently see those errors, however they occur from time to time.
In Heartbeat logs I do not see any errors. Those errors I see in Kibana Uptime section.
All of those monitors have the next pattern:
- type: http
schedule: '@every 2m'
username: someuser
password: somepassword
urls: ["https://someurl:someport/someuri"]
All of those instances are being resolved fine without delays.
My heartbeat instance is being run in Kubernetes cluster.
What could be the root of issue for these errors?
I use 6.7.2 version of all Elastic stack.