we are using a very simple one node ELK docker-compose based stack running on one server. Elasticsearch, Kibana, Logstash are all running version 5.6.3.
We use the persistent queue feature. I'm aware of the logstash issue with persistent queues and I think I made the required change already in the past.
Both the "old" (and restarted/running) Logstash 5.6 and new 6.6
config/logstash.yml (queue related content):
queue.type: persisted queue.drain: true path.queue: /usr/share/logstash/queue
What we do:
- copy 6.6 files to target directory
docker-compose down --volumes --remove-orphans(elasticsearch/logstash data is an external volume and will not be deleted)
docker-compose build --pull
docker-compose up -d
From what I read, the
queue.drain: true should force the queue to drain when stopping the container, right?
Still, after the
up the logstash keeps crashing/restarting:
[2019-03-08T13:36:56,607][INFO ][org.logstash.ackedqueue.QueueUpgrade] No PQ version file found, upgrading to PQ v2. [2019-03-08T13:36:56,861][ERROR][org.logstash.ackedqueue.QueueUpgrade] Logstash was unable to upgrade your persistent queue data.Please either downgrade to your previous version of Logstash and fully drain your persistent queue or delete your queue data.dir if you don't need to retain the data currently in your queue. [2019-03-08T13:36:56,877][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue. java.lang.IllegalStateException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Unexpected token (START_OBJECT), expected START_ARRAY: need JSON Array to contain As.WRAPPER_ARRAY type information for class java.util.HashMap ...
What am I missing? Is there a way to monitor the queue? Thank you in advance