Beats output to Logstash AND Elastic?

Elastic Stack Beats
, ,
1

Is it possible to Output to Elasticsearch AND Logstash at the same time from a Beat?

2

No, you can use only a single output. So you could send to Logstash and then have Logstash send to Elasticsearch.

3

But after receiving data in Logstash from beats, what i want is to send the output to different ES hosts(which is easy) but also send in on of them the event as it is, and to the other host i want to send a filtered event. How I'm gonna do that in Logstash pipeline?

4

You can use the pipeline output for this. Pipeline-to-Pipeline Communication | Logstash Reference [7.14] | Elastic

Create three pipelines. The first pipeline has the beats input and it outputs to the two other pipelines. In the other pipelines you can implement the independent filtering logic since the events are copied.

1 Like
5

Indeed Pipeline to Pipeline might save my problem, but i do have a question here. Is it mandatory to put everything into pipelines.yml? Because i have a big pipeline .conf file.

6

pipelines.yml only declares the pipelines. The config is still contained in .conf files.

#pipelines.yml
- pipeline.id: my-pipeline_1
  path.config: /etc/path/to/p1.conf
  pipeline.workers: 10
- pipeline.id: my-pipeline_2
  path.config: /etc/path/to/p2.conf
  pipeline.workers: 10

Docs: Multiple Pipelines | Logstash Reference [7.14] | Elastic

1 Like
7

Ok then, because i saw that in the docs of Pipeline to Pipeline, all the config was inside the pipelines.yml and i thought that was mandatory. Thanks.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.