We have a deployment in which different types of logs are getting pushed to Kafka. Then logstash is pulling these logs from Kafka and pushing it to ES. We want to prioritise the way logs getting indexed in ES. For example logs of
type A should be indexed faster than logs
type B. Basically if there are too many logs of
type B then they should not starve logs of
So can some one guide me if the below deployment will achieve this.
Create two topics in Kafka let's say
topic_B. Let's say
topic_A has 4 partitions and
topic_B has 2 partitions. Have two kafka input in logstash one reading from
topic_A with 4 consumer threads and the other one reading from
topic_B with two consumer threads. or have two logstash instance running one reading from
topic_A with 4 consumer threads and the other reading from
topic_B with 2 consumer threads.