Metricbeat unable to connect to elasticsearch behind proxy

Elastic Stack Beats
,
1

Hello,

I'm trying to connect Metricbeat from one cluster to Elasticsearch on another cluster. Elasticsearch is exposed via an Ingress (gateway) on a private domain.

I'm using ECK Operator version 2.12.1 for deployment. The ELK stack is on version 8.13.4.

My Metricbeat final config output looks like this:

output:
    elasticsearch:
        hosts:
            - https://elasticsearch.my-domain
            - https://elasticsearch.my-domain:443

The first host is injected by the ECK Operator, and I added the second one manually. For both hosts, I get connection errors. The logs are as follows:

{"log.level":"error","@timestamp":"2025-02-28T10:26:04.308Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:9200)): Get \"https://elasticsearch.my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}

and

{"log.level":"error","@timestamp":"2025-02-28T10:22:25.098Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:443)): 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}

In the first case, when I don't specify a port, it defaults to Elasticsearch's default port, which doesn't work since my proxy is listening on port 443.

In the second case, when I add port 443 to the hostname, I get a 404 error.

From inside the Metricbeat pod (and the browser as well), when I run:

root@ads-metricbeat-beat-metricbeat-v4vcs:/usr/share/metricbeat# curl -u user:pass https://elasticsearch.my-domain:443
{
  "name" : "ads-es-default-2",
  "cluster_name" : "ads",
  "cluster_uuid" : "FrOBtZxYQ3q3p36rGvllWQ",
  "version" : {
    "number" : "8.13.4",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "da95df118650b55a500dcc181889ac35c6d8da7c",
    "build_date" : "2024-05-06T22:04:45.107454559Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

For security reasons, I've replaced my real domain with my-domain.

Can you help me? Is this a bug, or am I doing something wrong?

2

Can you toggle debug logs logging.level: debug and share what prints out? That should help pinpoint which step is getting the 404.

Beats make 2 calls when connecting, a call to the root / and a call to /_license so that would be the other endpoint to check

3

Here are the logs:

{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.316Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:443)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.317Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://my-domain:9200)): Get \"https://my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":139},"message":"Attempting to reconnect to backoff(elasticsearch(https://my-domain:9200)) with 2 reconnect attempt(s)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:9200)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:42.577Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","network":"tcp","address":"my-domain:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.577Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: Get \"https://my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:45.318Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport.(*loggingConn).Read","file.name":"transport/logging.go","file.line":50},"message":"Error reading from connection: read tcp 10.222.1.249:37932->192.168.10.108:443: use of closed network connection","service.name":"metricbeat","network":"tcp","address":"my-domain:443","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://my-domain:443)): 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":139},"message":"Attempting to reconnect to backoff(elasticsearch(https://my-domain:443)) with 3 reconnect attempt(s)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:443)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.638Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":42},"message":"Completed dialing successfully","service.name":"metricbeat","network":"tcp","address":"my-domain:443","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.639Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
4

Looks like you're curling to https://elasticsearch.my-domain:443 but your metricbeat is targeting port 9200? can you share a log of metricbeat targeting 443?

5

Those logs are already present, since metricbeat is doing roundrobin between these two:

- https://elasticsearch.my-domain
- https://elasticsearch.my-domain:443

I'll share more of the configuration to give you better insight:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  annotations:
    association.k8s.elastic.co/es-conf: '{"authSecretName":"ads-metricbeat-elasticsearch","authSecretKey":"password","isServiceAccount":false,"caCertProvided":false,"caSecretName":"","url":"https://my-domain","version":"8.13.4"}'
    meta.helm.sh/release-name: elk
    meta.helm.sh/release-namespace: elk
  creationTimestamp: "2025-02-27T13:41:35Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ads-metricbeat
  namespace: elk
  resourceVersion: "19617528"
  uid: 99d6f5e1-6ef5-4786-be06-a1b10182eb0c
spec:
  elasticsearchRef:
    secretName: ads-metricbeat-elasticsearch
    ...

---

apiVersion: v1
kind: Secret
data:
  password:
  url: aHR0cHM6Ly9teS1kb21haW4K # https://my-domain  <-- without port
  username:
metadata:
  annotations:
    meta.helm.sh/release-name: elk
    meta.helm.sh/release-namespace: elk
  creationTimestamp: "2025-02-27T13:41:35Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ads-metricbeat-elasticsearch
  namespace: elk
  resourceVersion: "15928759"
  uid: bf200186-edcb-4565-b2a2-390aaeab06d6
type: Opaque

This is how I deploy Metricbeat. The key component is elasticsearchRef, which references a secret containing the Elasticsearch URL. The ECK operator deploys Metricbeat based on the provided configuration and injects the Elasticsearch URL into metricbeat.yaml—this is the first URL. From the operator’s perspective, everything works as expected since it also verifies connectivity.
The second URL, however, was manually inserted into metricbeat.yaml. As seen in the logs, Metricbeat attempts to connect to both URLs, but it automatically appends port 9200 to the first URL from the configuration (which originally had no port specified). I believe this is the issue—it shouldn't add a port if none was explicitly defined.

6

It looks like your curl was to a different domain? Your curl was to Elasticsearch.my-domain but your metricbeat is just connecting to my-domain?

7

Sorry, my bad. I accidentally replaced the whole domain with my-domain when I sent the previous post. That is not the issue.

8

Any updates on this?