Filebeat and Metricbeat get Error 401 Unauthorized

silentfilm · March 21, 2023, 10:19pm

I recently upgraded from ELK Stack 7.9.3 to 7.17.9. Everything is working great except that Filebeat and Metricbeat will not connect to Elasticsearch anymore unless they are installed on the same server. I get errors like these:

2023-03-21T13:55:51.425-0500    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500    INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500    DEBUG   [esclientleg]   eslegclient/connection.go:261   ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-21T13:55:51.427-0500    DEBUG   [esclientleg]   eslegclient/connection.go:265   Ping request failed with: 401 Unauthorized: Unauthorized

Testing filebeat on a server not running Elasticsearch:

[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: Unauthorized

Testing on the server that is running Elasticsearch:

[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.133.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.17.9

The issue is not the username or password because I can use curl to get Elasticsearch to respond:

[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

My filebeat configuration:

[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

name: filebeat

filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"

filebeat.config.inputs:
  enabled: true
  path: "/etc/filebeat/prospector-*.yml"
  reload.enabled: true
  reload.period: "30s"

processors:
  - add_cloud_metadata: ~
  - truncate_fields:
      fields:
        - log_event
      max_characters: 10000
      fail_on_error: false
      ignore_missing: true

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME}
  password: ${ES_FILEBEAT_PASSWORD}
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

logging.level: "debug"
logging.to_files: true
logging.files:
  path: "/var/log/filebeat"
  name: "filebeat.log"
  rotateeverybytes: 104857600
  interval: "24h"
  keepfiles: 7
  permissions: "0644"
logging.metrics:
  enalbled: true
  period: "60s"

setup.ilm.enabled: false
setup.template.enabled: false

I have xpack.security.enabled: false in elasticsearch.yml. What would cause these connection attempts to receive an unauthorized error message? As I said before, this same configuration worked in ES 7.9.3.

stephenb · March 22, 2023, 2:10am

Hi @silentfilm

A couple things to check...

Try the curl with the cert

curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200

silentfilm · March 22, 2023, 2:05pm

This curl command works well from all three servers:

[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
* About to connect() to aln-nbadev4.labs.server.com port 9200 (#0)
*   Trying 172.20.133.11...
* Connected to aln-nbadev4.labs.server.com (172.20.133.11) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/filebeat/cacert.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=esnode-aln-nbadev4.labs.server.com,OU=nBA,O=Server Systems Inc,L=City,ST=MA,C=US
*       start date: Mar 19 10:02:27 2023 GMT
*       expire date: Mar 17 10:02:27 2033 GMT
*       common name: esnode-aln-nbadev4.labs.server.com
*       issuer: CN=Default,O=Server,L=City,ST=Massachusetts,C=US
* Server auth using Basic with user 'filebeat_internal'
> GET / HTTP/1.1
> Authorization: Basic ZmlsZWJlYXRfaW50ZXJuYWw6bmV0c2NvdXQx
> User-Agent: curl/7.29.0
> Host: aln-nbadev4.labs.server.com:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 563
<
{
  "name" : "esnode-aln-nbadev4",
  "cluster_name" : "nba_elasticsearch_cluster",
  "cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
  "version" : {
    "number" : "7.17.9",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
    "build_date" : "2023-01-31T05:34:43.305517834Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host aln-nbadev4.labs.server.com left intact
[root@aln-nbav173-95 ~]#

@stephenb, You mentioned a couple of things to check, but only listed one command to try.

stephenb · March 22, 2023, 2:11pm

Yeah sorry I left that out. I know you want to use your filebeat user but try putting in the elastic user and password.

We want to rule out there's something different with the roles you've assigned. I know it works local and all those other things, but often when we get down to something obvious that should work, it's something obvious that's broken

silentfilm · March 22, 2023, 3:03pm

[root@aln-nbav173-95 ~]# curl -v --cacert /etc/filebeat/cacert.pem -u elastic -XGET https://aln-nbadev4.labs.server.com:9200

This same command works perfectly from all three servers using the elastic username and password.

stephenb · March 22, 2023, 3:37pm

Apologies I meant to put the elastic user in your filebeat.yml in the output.elasticsearch section on the box that does not work.

And try to start the beats

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

silentfilm · March 22, 2023, 4:46pm

Apologies I meant to put the elastic user in your filebeat.yml in the output.elasticsearch section on the box that does not work.
And try to start the beats

output.elasticsearch:
  enabled: true
  hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
  username: ${ES_FILEBEAT_USERNAME} <!--- HERE
  password: ${ES_FILEBEAT_PASSWORD} <!--- HERE
  protocol: "https"
  ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]

This was an excellent suggestion. With the elastic username and password hardcoded in filebeat.yml, filebeat was able to connect. So the issue is with my filebeat_internal user or the way that it is saved in the vault.

Thanks!

silentfilm · March 24, 2023, 1:10pm

It turned out that the Elasticsearch password was saved incorrectly in the filebeat (and metricbeat) keystore, so the issue was definitely just an authentication problem.

system · April 21, 2023, 3:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.