# Filebeat and Metricbeat get Error 401 Unauthorized

**URL:** <https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200>\
**Category:** Beats\
**Tags:** docker, filebeat, metricbeat\
**Created:** [March 21, 2023, 10:19pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200 "2023-03-21T22:19:04Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![silentfilm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silentfilm/32/44348_2.png) [@silentfilm](https://discuss.elastic.co/u/silentfilm)\
**Post date:** [March 21, 2023, 10:19pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/1 "2023-03-21T22:19:04Z")

</div>

I recently upgraded from ELK Stack 7.9.3 to 7.17.9. Everything is working great except that Filebeat and Metricbeat will not connect to Elasticsearch anymore unless they are installed on the same server. I get errors like these:

```auto
2023-03-21T13:55:51.425-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500 DEBUG [esclientleg] eslegclient/connection.go:261 ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500 INFO [publisher] pipeline/retry.go:223 done
2023-03-21T13:55:51.427-0500 DEBUG [esclientleg] eslegclient/connection.go:265 Ping request failed with: 401 Unauthorized: Unauthorized

```

Testing filebeat on a server not running Elasticsearch:

```auto
[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: Unauthorized

```

Testing on the server that is running Elasticsearch:

```auto
[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.9

```

The issue is not the username or password because I can use curl to get Elasticsearch to respond:

```auto
[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

My filebeat configuration:

```auto
[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

name: filebeat

filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"

filebeat.config.inputs:
  enabled: true
  path: "/etc/filebeat/prospector-*.yml"
  reload.enabled: true
  reload.period: "30s"

processors:
  - add_cloud_metadata: ~
  - truncate_fields:
      fields:
        - log_event
      max_characters: 10000
      fail_on_error: false
      ignore_missing: true

output.elasticsearch:
  enabled: true
  hosts: ["https://aln-nbadev4.labs.server.com:9200"]
  username: ${ES_FILEBEAT_USERNAME}
  password: ${ES_FILEBEAT_PASSWORD}
  protocol: "https"
  ssl.certificate_authorities: ["/etc/filebeat/cacert.pem"]

logging.level: "debug"
logging.to_files: true
logging.files:
  path: "/var/log/filebeat"
  name: "filebeat.log"
  rotateeverybytes: 104857600
  interval: "24h"
  keepfiles: 7
  permissions: "0644"
logging.metrics:
  enalbled: true
  period: "60s"

setup.ilm.enabled: false
setup.template.enabled: false

```

I have xpack.security.enabled: false in elasticsearch.yml. What would cause these connection attempts to receive an unauthorized error message? As I said before, this same configuration worked in ES 7.9.3.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 22, 2023, 2:10am UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/2 "2023-03-22T02:10:44Z")

</div>

Hi @silentfilm

> [@silentfilm](#):
>
> `curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200`

A couple things to check...

Try the curl with the cert

`curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200`

---

<div class="post-metadata">

**Author:** ![silentfilm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silentfilm/32/44348_2.png) [@silentfilm](https://discuss.elastic.co/u/silentfilm)\
**Post date:** [March 22, 2023, 2:05pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/3 "2023-03-22T14:05:48Z")

</div>

This curl command works well from all three servers:

```auto
[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
* About to connect() to aln-nbadev4.labs.server.com port 9200 (#0)
* Trying 172.20.133.11...
* Connected to aln-nbadev4.labs.server.com (172.20.133.11) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/filebeat/cacert.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=esnode-aln-nbadev4.labs.server.com,OU=nBA,O=Server Systems Inc,L=City,ST=MA,C=US
* start date: Mar 19 10:02:27 2023 GMT
* expire date: Mar 17 10:02:27 2033 GMT
* common name: esnode-aln-nbadev4.labs.server.com
* issuer: CN=Default,O=Server,L=City,ST=Massachusetts,C=US
* Server auth using Basic with user 'filebeat_internal'
> GET / HTTP/1.1
> Authorization: Basic ZmlsZWJlYXRfaW50ZXJuYWw6bmV0c2NvdXQx
> User-Agent: curl/7.29.0
> Host: aln-nbadev4.labs.server.com:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 563
<
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host aln-nbadev4.labs.server.com left intact
[root@aln-nbav173-95 ~]#

```

@stephenb, You mentioned a couple of things to check, but only listed one command to try.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 22, 2023, 2:11pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/4 "2023-03-22T14:11:54Z")

</div>

Yeah sorry I left that out. I know you want to use your filebeat user but try putting in the `elastic` user and password.

We want to rule out there's something different with the roles you've assigned. I know it works local and all those other things, but often when we get down to something obvious that should work, it's something obvious that's broken 🙂

---

<div class="post-metadata">

**Author:** ![silentfilm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silentfilm/32/44348_2.png) [@silentfilm](https://discuss.elastic.co/u/silentfilm)\
**Post date:** [March 22, 2023, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/5 "2023-03-22T15:03:36Z")

</div>

```auto
[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u elastic -XGET https://aln-nbadev4.labs.server.com:9200

```

This same command works perfectly from all three servers using the elastic username and password.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 22, 2023, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/6 "2023-03-22T15:37:08Z")

</div>

Apologies I meant to put the `elastic` user in your filebeat.yml in the `output.elasticsearch` section on the box that does not work.

And try to start the beats

```auto
output.elasticsearch:
  enabled: true
  hosts: ["https://aln-nbadev4.labs.server.com:9200"]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: ["/etc/filebeat/cacert.pem"]

```

---

<div class="post-metadata">

**Author:** ![silentfilm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silentfilm/32/44348_2.png) [@silentfilm](https://discuss.elastic.co/u/silentfilm)\
**Post date:** [March 22, 2023, 4:46pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/7 "2023-03-22T16:46:34Z")

</div>

> Apologies I meant to put the `elastic` user in your filebeat.yml in the `output.elasticsearch` section on the box that does not work.  
> And try to start the beats

```auto
output.elasticsearch:
  enabled: true
  hosts: ["https://aln-nbadev4.labs.server.com:9200"]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: ["/etc/filebeat/cacert.pem"]

```

This was an excellent suggestion. With the elastic username and password hardcoded in filebeat.yml, filebeat was able to connect. So the issue is with my filebeat\_internal user or the way that it is saved in the vault.

Thanks!

---

<div class="post-metadata">

**Author:** ![silentfilm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silentfilm/32/44348_2.png) [@silentfilm](https://discuss.elastic.co/u/silentfilm)\
**Post date:** [March 24, 2023, 1:10pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/8 "2023-03-24T13:10:20Z")

</div>

It turned out that the Elasticsearch password was saved incorrectly in the filebeat (and metricbeat) keystore, so the issue was definitely just an authentication problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2023, 3:10pm UTC](https://discuss.elastic.co/t/filebeat-and-metricbeat-get-error-401-unauthorized/328200/9 "2023-04-21T15:10:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.