Hi all,
Apologies if this is a really dumb question, but been reading so much think I am getting myself confused.
I have a filebeat agent running on a machine and its reporting back to my ELK stack server. When I had a single pipeline (main) with Logstash on the default port 5044 it worked really well.
I now have added multiple filebeat.yml's with different configs. When Filebeat starts up it loads all the configs. I noted in the documentation that you still have to have a base filebeat.yml to allow you to specify the conf.d with the additional files.
So I have:
filebeatdir >conf.d> filebeat_somelog.yml filebeat_someother.yml and filebeat_file.yml.
All is good. Filebeat starts up loads the configs and I can see it parsing the different inputs as specified in the different configs.
Question
Now I have multiple pipelines basically one for each of those configs. I am probably wrong but in each of the input files for logstash I specified a different port for each input. Thinking this is how that specific beat would talk to logstash. I am just lost on how to specifically tell the filebeat to go to a specific pipeline (the pipelines are being created correctly in elasticsearch).
What's happening now is all the filebeats outputs are ending up in the Main pipeline and ofcourse the filter for that pipeline is not filtering correctly.
So for example:
filebeat_somelog.yml > Pipe 1 (with Filter 1)
filebeat_someother.yml > Pipe 2 (with Filter 2)
filebeat_file.yml. > Pipe 3 (with Filter 3)
I am sure its probably obvious but you know when you read so many different things you get so confused? Would really appreciate the help.