Is it possible to Output to Elasticsearch AND Logstash at the same time from a Beat?
No, you can use only a single output. So you could send to Logstash and then have Logstash send to Elasticsearch.
But after receiving data in Logstash from beats, what i want is to send the output to different ES hosts(which is easy) but also send in on of them the event as it is, and to the other host i want to send a filtered event. How I'm gonna do that in Logstash pipeline?
You can use the pipeline output for this. Pipeline-to-Pipeline Communication | Logstash Reference [7.14] | Elastic
Create three pipelines. The first pipeline has the
beats input and it outputs to the two other pipelines. In the other pipelines you can implement the independent filtering logic since the events are copied.
Indeed Pipeline to Pipeline might save my problem, but i do have a question here. Is it mandatory to put everything into pipelines.yml? Because i have a big pipeline .conf file.
pipelines.yml only declares the pipelines. The config is still contained in .conf files.
#pipelines.yml - pipeline.id: my-pipeline_1 path.config: /etc/path/to/p1.conf pipeline.workers: 10 - pipeline.id: my-pipeline_2 path.config: /etc/path/to/p2.conf pipeline.workers: 10
Ok then, because i saw that in the docs of Pipeline to Pipeline, all the config was inside the pipelines.yml and i thought that was mandatory. Thanks.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.