I recently upgraded from ELK Stack 7.9.3 to 7.17.9. Everything is working great except that Filebeat and Metricbeat will not connect to Elasticsearch anymore unless they are installed on the same server. I get errors like these:
2023-03-21T13:55:51.425-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)): 401 Unauthorized: Unauthorized
2023-03-21T13:55:51.425-0500 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://aln-nbadev4.labs.server.com:9200)) with 1 reconnect attempt(s)
2023-03-21T13:55:51.425-0500 DEBUG [esclientleg] eslegclient/connection.go:261 ES Ping(url=https://aln-nbadev4.labs.server.com:9200)
2023-03-21T13:55:51.425-0500 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-21T13:55:51.426-0500 INFO [publisher] pipeline/retry.go:223 done
2023-03-21T13:55:51.427-0500 DEBUG [esclientleg] eslegclient/connection.go:265 Ping request failed with: 401 Unauthorized: Unauthorized
Testing filebeat on a server not running Elasticsearch:
[root@aln-nbav173-95 filebeat]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.20.133.11
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... ERROR 401 Unauthorized: Unauthorized
Testing on the server that is running Elasticsearch:
[root@aln-nbadev4 elasticsearch]# filebeat test output
elasticsearch: https://aln-nbadev4.labs.server.com:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.20.133.11
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.17.9
The issue is not the username or password because I can use curl to get Elasticsearch to respond:
[root@aln-nbav173-95 filebeat]# curl -k -u filebeat_internal -XGET https://aln-nbadev4.labs.server.com:9200
Enter host password for user 'filebeat_internal':
{
"name" : "esnode-aln-nbadev4",
"cluster_name" : "nba_elasticsearch_cluster",
"cluster_uuid" : "mkT5taWDRGOaDPXOMo5_5g",
"version" : {
"number" : "7.17.9",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "ef48222227ee6b9e70e502f0f0daa52435ee634d",
"build_date" : "2023-01-31T05:34:43.305517834Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
My filebeat configuration:
[root@aln-nbadev4 ~]# more /etc/filebeat/filebeat.yml
---
# Filebeat Configuration
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
name: filebeat
filebeat.shutdown_timeout: "5s"
filebeat.registry.path: "/var/lib/filebeat/registry"
filebeat.registry.file_permissions: "0600"
filebeat.registry.flush: "0s"
filebeat.config.inputs:
enabled: true
path: "/etc/filebeat/prospector-*.yml"
reload.enabled: true
reload.period: "30s"
processors:
- add_cloud_metadata: ~
- truncate_fields:
fields:
- log_event
max_characters: 10000
fail_on_error: false
ignore_missing: true
output.elasticsearch:
enabled: true
hosts: [ "https://aln-nbadev4.labs.server.com:9200" ]
username: ${ES_FILEBEAT_USERNAME}
password: ${ES_FILEBEAT_PASSWORD}
protocol: "https"
ssl.certificate_authorities: [ "/etc/filebeat/cacert.pem" ]
logging.level: "debug"
logging.to_files: true
logging.files:
path: "/var/log/filebeat"
name: "filebeat.log"
rotateeverybytes: 104857600
interval: "24h"
keepfiles: 7
permissions: "0644"
logging.metrics:
enalbled: true
period: "60s"
setup.ilm.enabled: false
setup.template.enabled: false
I have xpack.security.enabled: false in elasticsearch.yml. What would cause these connection attempts to receive an unauthorized error message? As I said before, this same configuration worked in ES 7.9.3.