I am getting the following error, and after a few hours of reasearch and configuration changes I could not find a solution.
Jul 26 00:31:05 am01 metricbeat: {"level":"error","timestamp":"2019-07-26T00:31:05.712Z","caller":"pipeline/output.go:100","message":"Failed to connect to backoff(elasticsearch(https://elasticsearch-url:elasticsearchport)): Connection marked as failed because the onConnect callback failed: 503 Service Unavailable: {"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-metricbeat-7.2.0) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (put-lifecycle-metricbeat-7.2.0) within 30s"},"status":503}"}
the error suggest that the metricbeat cannot connect to elasticsearch/kibana, but if I try from the same host a wget to that url I get the following response:
{
"name" : "elasticsearch-server-hostname",
"cluster_name" : "logging-prod",
"cluster_uuid" : "dJpkSiwlTj-QN2KGIcaTpQ",
"version" : {
"number" : "7.2.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "508c38a",
"build_date" : "2019-06-20T15:54:18.811730Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
What could be the problem?
Thank you