Hi,
I am quite new to the ELK stack and would appreciate some guidance on the brief mentioned in the subject.
My goal is to have multiple providers (nginx1, nginx2) broadcasting beats events with hints enabled to filebeat deployed as a DaemonSet. filebeat should then forward the logs to a single instance of logstash. Depending on the provider that sent the logs, logstash should then forward those to a specific instance of elastic (elastic1 or elastic2).
So I managed to get this working as a single pipeline i.e.
nginx1 - filebeat - logstash - elastic1
I am a bit stuck on how to make filebeat distinguish between the providers that send the beat events. I have configured nginx1 and nginx2 with the following annotations:
annotations:
co.elastic.logs/enabled: "true"
co.elastic.logs/module: nginx
co.elastic.logs/fileset.stdout: access
co.elastic.logs/fileset.stderr: error
Is there a custom label or tag that I can add to this list of annotations which can then be referenced in logstash? I know that an IF...ELSE statement can be used in the logstash configuration within the output block, but am not sure how to bring it all together.
Any advice would be much appreciated.
Thanks,