Filters in a Distributed Logstash Setup

Elastic Stack Logstash
1

I've recently inherited the following setup to manage:

Logstash (Collector) -> Redis -> Logstash Indexer -> Elasticsearch

I can see a number of filters on the collector instances as well as the indexers. Am I correct in thinking these filters should all be on the indexer so the collectors simply receive logs on an input and push them into Redis as an output?

Am I likely to break anything if I merge the collector filters into the indexer filters, some of which I can see is duplicated at present?

2

I can see a number of filters on the collector instances as well as the indexers. Am I correct in thinking these filters should all be on the indexer so the collectors simply receive logs on an input and push them into Redis as an output?

Yes, I prefer keeping the leaf instances dumb and doing as much filtering as possible on the central instances.

Am I likely to break anything if I merge the collector filters into the indexer filters, some of which I can see is duplicated at present?

I think there's a good chance that you'll be okay, but it obviously depends on your filters.

Shameless plug: This could be a good excuse for looking into GitHub - magnusbaeck/logstash-filter-verifier as a way of testing your full set of filters.

3

Perfect thank you for the advice.

Certainly seems to answer a lot of my questions - I'll check it out.