Questions about DNS filter and multiple pipelines

Elastic Stack Logstash
1

Hello to everyone!
I plan to use Logstash to transform syslog messages into info files
To achieve, it I decided to use multi-pipeline configuration:
pipelines.yml

- pipeline.id: ans_file_syslog
  path.config: "C:/Program Files/logstash/etc/pipelines/ans_file_syslog.conf"

- pipeline.id: asa_file_syslog
  path.config: "C:/Program Files/logstash/etc/pipelines/asa_file_syslog.conf"

- pipeline.id: asr_file_syslog
  path.config: "C:/Program Files/logstash/etc/pipelines/asr_file_syslog.conf"

For example, one of *.conf file looks like this:
asa_file_syslog.conf

input {
    udp {
        port => 1114
    }
}

filter {
    dns {
        action => "replace"
        hit_cache_size => 1024
        reverse => [ "[host][ip]"]
    }
}

output {
    file {
        codec => line { format => "%{message}"}
        path => "E:/logstash/asa/asa_file_syslog/%{[host][ip]}/%{+YYYY-MM-dd-HH}.log"
    }
}

Because of a necessity to use a DNS filter in each *.conf file, I have a couple of questions:

  1. Is it possible to use only one filter that is defined only once? If yes, how?
    I read about pipeline-to-pipeline communication but didn't grasp how I can return a data flow to the original pipe

  2. Does it make sense to use one filter instead of defining it in each file? Except for getting rid of code reuse

  3. How can I define a global value for hit_cache_size that will be used across all pipelines?
    I understand that I can use a small value for each pipe, but it seems a little bit excessive

2

No, if you use multiple pipelines, then you will need one dns filter per pipeline.

If you use pipeline-to-pipeline you cannot return the data to the original pipeline, you would need to redirect it to another pipeline.

You could use a common pipeline with a common filter, but this would complicated a little your ingestion as you would need an extra pipeline for it one of your pipelines and use conditionals in the output of the common pipeline to redirect the input to the correct output.

I do not see any reason, why not use a dns filter in each one of the pipelines?

That's not possible, the configuration is per filter.

If your pipelines are that simple, you could just use one pipeline and use conditionals in the output.

3

Great thanks to you! =)