Filebeat and Metricbeat get Error 401 Unauthorized

I recently upgraded from ELK Stack 7.9.3 to 7.17.9. Everything is working great except that Filebeat and Metricbeat will not connect to Elasticsearch anymore unless they are installed on the same server. I get errors like these:

2023-03-21T13:55:51.425-0500    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500    INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500    DEBUG   [esclientleg]   eslegclient/connection.go:261   ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-21T13:55:51.427-0500    DEBUG   [esclientleg]   eslegclient/connection.go:265   Ping request failed with: 401 Unauthorized: Unauthorized

Testing filebeat on a server not running Elasticsearch:

[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: Unauthorized

Testing on the server that is running Elasticsearch:

[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.9

The issue is not the username or password because I can use curl to get Elasticsearch to respond:

[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

My filebeat configuration:

[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

name: filebeat

filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"

filebeat.config.inputs:
  enabled: true
  path: "/etc/filebeat/prospector-*.yml"
  reload.enabled: true
  reload.period: "30s"

processors:
  - add_cloud_metadata: ~
  - truncate_fields:
      fields:
        - log_event
      max_characters: 10000
      fail_on_error: false
      ignore_missing: true

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME}
  password: ${ES_FILEBEAT_PASSWORD}
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

logging.level: "debug"
logging.to_files: true
logging.files:
  path: "/var/log/filebeat"
  name: "filebeat.log"
  rotateeverybytes: 104857600
  interval: "24h"
  keepfiles: 7
  permissions: "0644"
logging.metrics:
  enalbled: true
  period: "60s"

setup.ilm.enabled: false
setup.template.enabled: false

I have xpack.security.enabled: false in elasticsearch.yml. What would cause these connection attempts to receive an unauthorized error message? As I said before, this same configuration worked in ES 7.9.3.

Hi @silentfilm

A couple things to check...

Try the curl with the cert

curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200

This curl command works well from all three servers:

[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
* About to connect() to aln-nbadev4.labs.server.com port 9200 (#0)
*   Trying 172.20.133.11...
* Connected to aln-nbadev4.labs.server.com (172.20.133.11) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/filebeat/cacert.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=esnode-aln-nbadev4.labs.server.com,OU=nBA,O=Server Systems Inc,L=City,ST=MA,C=US
*       start date: Mar 19 10:02:27 2023 GMT
*       expire date: Mar 17 10:02:27 2033 GMT
*       common name: esnode-aln-nbadev4.labs.server.com
*       issuer: CN=Default,O=Server,L=City,ST=Massachusetts,C=US
* Server auth using Basic with user 'filebeat_internal'
> GET / HTTP/1.1
> Authorization: Basic ZmlsZWJlYXRfaW50ZXJuYWw6bmV0c2NvdXQx
> User-Agent: curl/7.29.0
> Host: aln-nbadev4.labs.server.com:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 563
<
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host aln-nbadev4.labs.server.com left intact
[root@aln-nbav173-95 ~]#

@stephenb, You mentioned a couple of things to check, but only listed one command to try.

Yeah sorry I left that out. I know you want to use your filebeat user but try putting in the elastic user and password.

We want to rule out there's something different with the roles you've assigned. I know it works local and all those other things, but often when we get down to something obvious that should work, it's something obvious that's broken :slight_smile:

[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u elastic -XGET https://aln-nbadev4.labs.server.com:9200

This same command works perfectly from all three servers using the elastic username and password.

Apologies I meant to put the elastic user in your filebeat.yml in the output.elasticsearch section on the box that does not work.

And try to start the beats

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]
1 Like

Apologies I meant to put the elastic user in your filebeat.yml in the output.elasticsearch section on the box that does not work.
And try to start the beats

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

This was an excellent suggestion. With the elastic username and password hardcoded in filebeat.yml, filebeat was able to connect. So the issue is with my filebeat_internal user or the way that it is saved in the vault.

Thanks!

It turned out that the Elasticsearch password was saved incorrectly in the filebeat (and metricbeat) keystore, so the issue was definitely just an authentication problem.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.