Hello,
I'm trying to connect Metricbeat from one cluster to Elasticsearch on another cluster. Elasticsearch is exposed via an Ingress (gateway) on a private domain.
I'm using ECK Operator version 2.12.1 for deployment. The ELK stack is on version 8.13.4.
My Metricbeat final config output looks like this:
output:
elasticsearch:
hosts:
- https://elasticsearch.my-domain
- https://elasticsearch.my-domain:443
The first host is injected by the ECK Operator, and I added the second one manually. For both hosts, I get connection errors. The logs are as follows:
{"log.level":"error","@timestamp":"2025-02-28T10:26:04.308Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:9200)): Get \"https://elasticsearch.my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
and
{"log.level":"error","@timestamp":"2025-02-28T10:22:25.098Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:443)): 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
In the first case, when I don't specify a port, it defaults to Elasticsearch's default port, which doesn't work since my proxy is listening on port 443.
In the second case, when I add port 443 to the hostname, I get a 404 error.
From inside the Metricbeat pod (and the browser as well), when I run:
root@ads-metricbeat-beat-metricbeat-v4vcs:/usr/share/metricbeat# curl -u user:pass https://elasticsearch.my-domain:443
{
"name" : "ads-es-default-2",
"cluster_name" : "ads",
"cluster_uuid" : "FrOBtZxYQ3q3p36rGvllWQ",
"version" : {
"number" : "8.13.4",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "da95df118650b55a500dcc181889ac35c6d8da7c",
"build_date" : "2024-05-06T22:04:45.107454559Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
For security reasons, I've replaced my real domain with my-domain
.
Can you help me? Is this a bug, or am I doing something wrong?