I didn't try but it should work. I don't see any issue as well as the logic to read twice the same file
Maybe is not good idea to have in the same index, you will have duplicated data. Or if you like to read a file and write to 2 indices, then set 2 ES index in the output.
Thank you!
I also don't see an issue, but i want to be sure.
Currently i am using only one pipeline which uses "clone" filter plugin and then routing to different outputs, but for some reason it is killing the CPU on the machine. I was able to pinpoint that the clone plugin is the problem.
So my next idea is to avoid using the clone plugin, and seprate this in two piplines
How did you arrive at this conclusion? Can you share your config for the clone filter?
The clone plugin is pretty simple, I would not expect it to have any impact in performance.
Reading the same file with 2 inputs and using 2 different sincedb path should work, but another alternative is to read the pipeline once, and then use the pipeline-to-pipeline communication to send it to the two processing pipelines.
In this case you would have 3 pipelines, one reading the file and 2 processing it.
Basically in the filter phase i am just deciding where to send the data based on a @metadata field.
The i just route the out based on the @metadata field
It is probably worth saying that the logstash version is .. ancient (2.3.3). I need to use such an old version becuse of Kafaka compability issues with our very old version of Kafka (0.9)
It's not "old", it's from the time of dinosaurs and reptiles.
Can you change logic in the output? The clone makes 2 same message, but IF again route to single destination. In case you always need to send to Kafka, then use only if on rabbitmq.
Are you doing any other transformation or this is you entire pipeline?
I'm not sure why you are cloning the events, the only difference between them is that for the kafka output you have an extra conditional, why not have this conditional on the output?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.