Filebeat EKS - sidecar container

Elastic Stack Beats
1

Hi,

I have a 3rd party image deployed in EKS, and it writes a lot of logs in different files. Now I have to push the logs from different files generated by the product into ES and viewed through Kibana

I am using sidecar container with filebeat image so that it can collect the logs and push it to Elastic Search. In the EKS, fluentd daemonset has been configured, so if you put it in the console then the logs are pushed o ES.

I designed the pipeline as below -
Log Files -> FileBeat -> Console Output -> FLuentD -> ElasticSearch -> Kibana

Its working and we are receiving the events in Kibana.

But I have to modify the filebeat configuration to achieve the below items -

  1. In Filebeat configuration as I have used *.log so I have to either add a custom tag (may be with the filename) so that we can easily identify the event. If we add the custom tag and since the filebeat outputs to console will the tag be propagated to the ES.
    Please note that the fluentD and ES are enterprise installation and we can't change it.

  2. Remove the filebeat log/harvester logs from going to ES

Thanks and Regards
Samaresh

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.