# Metricbeat unable to connect to elasticsearch behind proxy

**URL:** <https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204>\
**Category:** Beats\
**Tags:** docker, metricbeat\
**Created:** [February 28, 2025, 10:51am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204 "2025-02-28T10:51:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rade\_Pajic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rade_pajic/32/122949_2.png) [@Rade\_Pajic](https://discuss.elastic.co/u/Rade_Pajic)\
**Post date:** [February 28, 2025, 10:51am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/1 "2025-02-28T10:51:43Z")

</div>

Hello,

I'm trying to connect Metricbeat from one cluster to Elasticsearch on another cluster. Elasticsearch is exposed via an Ingress (gateway) on a private domain.

I'm using ECK Operator version 2.12.1 for deployment. The ELK stack is on version 8.13.4.

My Metricbeat final config output looks like this:

```auto
output:
    elasticsearch:
        hosts:
            - https://elasticsearch.my-domain
            - https://elasticsearch.my-domain:443

```

The first host is injected by the ECK Operator, and I added the second one manually. For both hosts, I get connection errors. The logs are as follows:

```auto
{"log.level":"error","@timestamp":"2025-02-28T10:26:04.308Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:9200)): Get \"https://elasticsearch.my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}

```

and

```auto
{"log.level":"error","@timestamp":"2025-02-28T10:22:25.098Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.my-domain:443)): 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}

```

In the first case, when I don't specify a port, it defaults to Elasticsearch's default port, which doesn't work since my proxy is listening on port 443.

In the second case, when I add port 443 to the hostname, I get a 404 error.

From inside the Metricbeat pod (and the browser as well), when I run:

```auto
root@ads-metricbeat-beat-metricbeat-v4vcs:/usr/share/metricbeat# curl -u user:pass https://elasticsearch.my-domain:443
{
  "name" : "ads-es-default-2",
  "cluster_name" : "ads",
  "cluster_uuid" : "FrOBtZxYQ3q3p36rGvllWQ",
  "version" : {
    "number" : "8.13.4",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "da95df118650b55a500dcc181889ac35c6d8da7c",
    "build_date" : "2024-05-06T22:04:45.107454559Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

```

For security reasons, I've replaced my real domain with `my-domain`.

Can you help me? Is this a bug, or am I doing something wrong?

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 1, 2025, 2:51am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/2 "2025-03-01T02:51:00Z")

</div>

Can you toggle debug logs `logging.level: debug` and share what prints out? That should help pinpoint which step is getting the 404.

Beats make 2 calls when connecting, a call to the root `/` and a call to `/_license` so that would be the other endpoint to check

---

<div class="post-metadata">

**Author:** ![Rade\_Pajic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rade_pajic/32/122949_2.png) [@Rade\_Pajic](https://discuss.elastic.co/u/Rade_Pajic)\
**Post date:** [March 3, 2025, 5:39pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/3 "2025-03-03T17:39:18Z")

</div>

Here are the logs:

```auto
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.316Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:443)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.317Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://my-domain:9200)): Get \"https://my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":139},"message":"Attempting to reconnect to backoff(elasticsearch(https://my-domain:9200)) with 2 reconnect attempt(s)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.575Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:9200)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:42.577Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","network":"tcp","address":"my-domain:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:42.577Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: Get \"https://my-domain:9200\": dial tcp 192.168.10.108:9200: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:45.318Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport.(*loggingConn).Read","file.name":"transport/logging.go","file.line":50},"message":"Error reading from connection: read tcp 10.222.1.249:37932->192.168.10.108:443: use of closed network connection","service.name":"metricbeat","network":"tcp","address":"my-domain:443","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://my-domain:443)): 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":139},"message":"Attempting to reconnect to backoff(elasticsearch(https://my-domain:443)) with 3 reconnect attempt(s)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.633Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":284},"message":"ES Ping(url=https://my-domain:443)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.638Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":42},"message":"Completed dialing successfully","service.name":"metricbeat","network":"tcp","address":"my-domain:443","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-03-03T17:30:46.639Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":288},"message":"Ping request failed with: 404 Not Found: ","service.name":"metricbeat","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 3, 2025, 8:34pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/4 "2025-03-03T20:34:16Z")

</div>

Looks like you're curling to `https://elasticsearch.my-domain:443` but your metricbeat is targeting port `9200`? can you share a log of metricbeat targeting 443?

---

<div class="post-metadata">

**Author:** ![Rade\_Pajic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rade_pajic/32/122949_2.png) [@Rade\_Pajic](https://discuss.elastic.co/u/Rade_Pajic)\
**Post date:** [March 3, 2025, 9:24pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/5 "2025-03-03T21:24:01Z")

</div>

Those logs are already present, since metricbeat is doing roundrobin between these two:

```auto
- https://elasticsearch.my-domain
- https://elasticsearch.my-domain:443

```

I'll share more of the configuration to give you better insight:

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  annotations:
    association.k8s.elastic.co/es-conf: '{"authSecretName":"ads-metricbeat-elasticsearch","authSecretKey":"password","isServiceAccount":false,"caCertProvided":false,"caSecretName":"","url":"https://my-domain","version":"8.13.4"}'
    meta.helm.sh/release-name: elk
    meta.helm.sh/release-namespace: elk
  creationTimestamp: "2025-02-27T13:41:35Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ads-metricbeat
  namespace: elk
  resourceVersion: "19617528"
  uid: 99d6f5e1-6ef5-4786-be06-a1b10182eb0c
spec:
  elasticsearchRef:
    secretName: ads-metricbeat-elasticsearch
    ...

---

apiVersion: v1
kind: Secret
data:
  password:
  url: aHR0cHM6Ly9teS1kb21haW4K # https://my-domain <-- without port
  username:
metadata:
  annotations:
    meta.helm.sh/release-name: elk
    meta.helm.sh/release-namespace: elk
  creationTimestamp: "2025-02-27T13:41:35Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ads-metricbeat-elasticsearch
  namespace: elk
  resourceVersion: "15928759"
  uid: bf200186-edcb-4565-b2a2-390aaeab06d6
type: Opaque

```

This is how I deploy Metricbeat. The key component is `elasticsearchRef`, which references a secret containing the Elasticsearch URL. The ECK operator deploys Metricbeat based on the provided configuration and injects the Elasticsearch URL into `metricbeat.yaml`—this is the first URL. From the operator’s perspective, everything works as expected since it also verifies connectivity.  
The second URL, however, was manually inserted into `metricbeat.yaml`. As seen in the logs, Metricbeat attempts to connect to both URLs, but it automatically appends port `9200` to the first URL from the configuration (which originally had no port specified). I believe this is the issue—it shouldn't add a port if none was explicitly defined.

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 4, 2025, 5:40am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/6 "2025-03-04T05:40:06Z")

</div>

It looks like your curl was to a different domain? Your curl was to Elasticsearch.my-domain but your metricbeat is just connecting to my-domain?

---

<div class="post-metadata">

**Author:** ![Rade\_Pajic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rade_pajic/32/122949_2.png) [@Rade\_Pajic](https://discuss.elastic.co/u/Rade_Pajic)\
**Post date:** [March 4, 2025, 6:26pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/7 "2025-03-04T18:26:15Z")

</div>

Sorry, my bad. I accidentally replaced the whole domain with `my-domain` when I sent the previous post. That is not the issue.

---

<div class="post-metadata">

**Author:** ![Rade\_Pajic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rade_pajic/32/122949_2.png) [@Rade\_Pajic](https://discuss.elastic.co/u/Rade_Pajic)\
**Post date:** [March 10, 2025, 3:03pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-connect-to-elasticsearch-behind-proxy/375204/8 "2025-03-10T15:03:23Z")

</div>

Any updates on this?