Exiting: couldn't connect to any of the configured Elasticsearch hosts. 503 Service Unavailable: <html><body><h1>503 Service Unavailable</h1>

Elastic Observability
1

I have an ELK stack (8.17.0) configured with ECK operator in my on premise K8s Cluster. I have a web server with apache, and im trying to visualize the apache logs on Kibana. But the filebeats is showing 503 service unavailable error.

filebeat setup

Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://elasticsearch.internal.clockhash.com:443: 503 Service Unavailable:

503 Service Unavailable


No server is available to handle this request.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://elasticsearch.internal.clockhash.com"]
  ssl.verification_mode: "none"
  # Performance preset - one of "balanced", "throughput", "scale",
  # "latency", or "custom".
  #preset: balanced

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "<username>"
  password: <password>

I have created an ingress (elasticsearch.internal.clockhash.com) for exposing the elastic to outside to get access from AWS ec2.

Curl command works fine.

root@ip-172-31-24-112:~# curl -X GET "https://elasticsearch.internal.clockhash.com/_cat/indices?v" -u <username>:<Password> --insecure
health status index                                                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   .internal.alerts-observability.logs.alerts-default-000001          sR5Bs6NsSHmNPMUsTDhe6w   1   1          0            0       500b
 250b         250b
green  open   filebeat-logs
2

Hi @Ranjith_Kumar Welcome to the community.

Is that curl from the same host /container filebeat is running on?

Try explicitly setting the port?
hosts: ["https://elasticsearch.internal.clockhash.com:443"]

because filebeat will default the elasticsearch port to 9200

Are you running filebeat directly on the host or in a container?

Did you turn on monitoring in filebeat.yml? (always best to share the whole thing instead of snippets, because filebeat monitoring itself will also try to connect)