Filebeat not connecting to elasticsearch cluster running with ECK

[aksh_18 (Akash)]

Hi All,

I have deployed ES cluster and kibana with the help of ECK. I am using filebeat-kubernetes.yaml to deploy filebeat. I deployed metricbeat with metricbeat-kubernetes.yaml and its running fine. But even after applying same configuration to filebeat yaml it is not able to connect to ES. Following are the logs related TLS and ES url. Please help on this.

2021-07-08T11:47:43.229Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.13.2' as ILM is enabled.
2021-07-08T11:47:43.229Z        WARN    [cfgwarn]       tlscommon/config.go:105 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-07-08T11:47:43.229Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://quickstart-es-http:9200
2021-07-08T11:47:43.229Z        INFO    [publisher]     pipeline/module.go:113  Beat name: oke-cgusmlcotva-nti7iapgfvq-sl4fjm436ua-2
2021-07-08T11:47:43.230Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-07-08T11:47:43.230Z        INFO    instance/beat.go:473    filebeat start running.
2021-07-08T11:47:43.231Z        INFO    memlog/store.go:119     Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-07-08T11:47:43.231Z        INFO    memlog/store.go:124     Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0

[abrx]

Did you configure the use of password and TLS in order to index data ?

I suggest you to put them in the same namespace if possible in order to:

• use the elastic ServiceAccount created by ECK in order to ingest data
• mount the secret carrying the ES certificate in your beats pods
• maybe adress ES directly via its service, using <service_name>.<namespace> url which is known by the pods inside the namespace, so you avoid loading your IngressManager and bypass any authentification you put in it (you still have to use a valide ES user to ingest/read data)

[aksh_18 (Akash)]

Thanks for the reply @abrx

There was problem regarding harvesting of the log files. Filebeat was not able to harvest the data from the given docker log file paths. But when I uncommented autodiscover section of the filebeat-kubernetes.yaml , it started ingesting logs into elasticsearch.