Is it possible to setup multiple logstash indexer service for same index?

Elastic Stack Logstash
1

we have 2 index which has huge data feeding. the forwarder sends data to redis then indexer.

by checking the LLEN of the KEYS in redis-cli, the number keeps increasing which means the indexer is not picking up the file fast enough. so I am thinking to setup another indexer service handling the same index.

is this possible??

2

Two or more Logstash services, each with one redis input? Sure, you can do that. Just make sure you don't have any filters that depend on state between messages (e.g. multiline, elapsed, aggregate, collate).

Of course, this will only help if Logstash indeed is the bottleneck. If ES or some other downstream service is what's holding you up it won't make a difference.

3

Hi Magnus, thanks for your reply.
each with one redis input? do you mean each with same redis input?

Do you know how does these 2 threads coordinate with each other? is it load balanced? or will some message be double fed or even missed?

4

each with one redis input? do you mean each with same redis input?

That's another way of putting it. They have one redis input each, both with identical configuration.

Do you know how does these 2 threads coordinate with each other? is it load balanced? or will some message be double fed or even missed?

Those are really Redis questions. Redis guarantees that a message is fetched exactly once. I'd assume that messages are fetched via round robin between clients or something sufficiently close to round robin.