I have an index X which continuosly receives maschine logs. I need to reindex the data constantly to index Y which is connected to Kibana. I have a Elasticsearch input, filter and Elasticsearch output. The reindexing works fine but after all documents have been reindexed, Logstash automatically shuts down because probably one of the elasticsearch plugins sends out a shut down command.
However, index X still receives logs which are then not stashed into index Y and in consequence cannot be seen in Kibana. I don't think it's a problem of my configuration, it's just the way the Elasticsearch plugin is programmed. I know a possible solution would be an exec plugin which starts the pipeline again and again in a given interval. But since the pipeline is starting and shutting down every time this would be very ugly and not ressource friendly.
Is there any way to keep the pipeline with an Elasticsearch Input alive and not shutting it down automatically?
Thanks in advance