Filebeat not working with xpack Security enabled in Elasticsearch

Elastic Stack Beats
1

Hi,

We are running a 3 node elastic cluster to index logs from a firewall.

For monitoring the cluster health, we are trying to collect the elastic cluster logs through filebeat and visualize these logs in kibana. For this we installed filebeat in the one of the nodes within the cluster.

The configuration of filebeat.yml is as follows:

filebeat.inputs:

- type: filestream

  id: my-filestream-id

  enabled: true

  paths:
    - /var/log/*.log

# ---------------------------- Elasticsearch Output -----------------------------
output.elasticsearch:
  hosts: ["https://192.168.1.6:9200","https://192.168.1.20:9200","https://192.168.1.21:9200"]

  preset: balanced

  protocol: "https"

  username: "username"
  password: "PASSWORD"
  ssl:
    enabled: true
#this will be replaced using secret keystore once it works
    ca_trusted_fingerprint: "22d0ddcf89079acec7ef398d4cb658c2ec560ece393f03ffede55c84692f73e1"
    certificate_authorities: "/etc/elasticsearch/certs/elastic-stack-ca.pem"


# =================================== Kibana ===================================

setup.kibana:

  host: "https://192.168.1.6:5601"
  username: "username"
  password: "PASSWORD"

  ssl:
  enabled: true
  certificate_authorities: "/etc/kibana/elastic-stack-ca.pem"
  certificate: "/etc/kibana/kibana-server.pem"
  key: "/etc/kibana/kibana-server.key"

We also enabled the elasticsearch module in modules.d in Filebeat and the configuration of elasticsearch.yml is as follows:

- module: elasticsearch
  # Server log
  server:
    enabled: true

    var.paths:
      - /var/log/elasticsearch/*.json
      - /var/log/elasticsearch/*_server.json

While settingup Filebeat by running following command: " filebeat setup -e " we are getting the errors mentioned below:

{"log.level":"error","@timestamp":"2024-04-23T16:15:54.092+0530","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: received error failed requesting azure metadata: Get \"http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01\": dial tcp 169.254.169.254:80: i/o timeout","service.name":"filebeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-04-23T16:17:21.102+0530","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnectedClient","file.name":"eslegclient/connection.go","file.line":252},"message":"error connecting to Elasticsearch at https://192.168.1.6:9200: Get \"https://192.168.1.6:9200\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","service.name":"filebeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-04-23T16:17:21.102+0530","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp 192.168.1.6:9200: i/o timeout","service.name":"filebeat","network":"tcp","address":"192.168.1.6:9200","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-04-23T16:17:21.904+0530","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1340},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://192.168.1.6:5601/api/status fails: fail to execute the HTTP GET request: Get \"https://192.168.1.6:5601/api/status\": x509: certificate signed by unknown authority (status=0). Response: ","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://192.168.1.6:5601/api/status fails: fail to execute the HTTP GET request: Get "https://192.168.1.6:5601/api/status": x509: certificate signed by unknown authority (status=0). Response:

We are using xpack security with self signed certificates. The error appears to be related to the use of a self signed certificate.

How can this be resolved?