Our current architecture for indexing is: we have a kafka topic from which logstash reads and then indexes in elastic.
We sometimes have peaks of indexing which is causing search issues. I would like to be able to slow down the indexing speed based on elastic search metrics, is that possible?
That's really an elasticsearch question. If elasticsearch indexing slows down there will be back-pressure that will stop logstash sending it more data. It has been answered here. Not sure if things have changed. I do not run elasticsearch.