I have tried to look into similar threads, but I am not sure I got to a conclusive solution.
Here is my situation:
I have a number N of log files, with same type of content
I have already a LogStash config file working fine when FileBeat reads one of those logs. As I need to process different lines from the logs at once, the filter section in LogStash is a piece of code in Ruby. Aggregate filter didn't work, for example.
Now, if I try to read the N logs with FileBeat at once, the lines from all of them get mixed, and my Ruby logic does not work anymore.
I could have a different LogStash pipeline for each input log, if needed. I understand each one of them would have to listen to a different port, correct?
What I don't see is how to configure FileBeat to send the data from each input log into a different port, so it can be processed by a different LogStash's pipeline.
In a nutshell:
- FileBeat reads log1 and sends it to port #1 for LogStash pipeline #1
- FileBeat reads log2 and sends it to port #2 for LogStash pipeline #2
Is it possible? Makes sense?
I read some people solved it by running multiple Filebeat instances at once. Is that the only way?
Thanks a lot in advance.