Hi there,
we are using a very simple one node ELK docker-compose based stack running on one server. Elasticsearch, Kibana, Logstash are all running version 5.6.3.
We use the persistent queue feature. I'm aware of the logstash issue with persistent queues and I think I made the required change already in the past.
Both the "old" (and restarted/running) Logstash 5.6 and new 6.6 config/logstash.yml (queue related content):
queue.type: persisted
queue.drain: true
path.queue: /usr/share/logstash/queue
What we do:
- copy 6.6 files to target directory
-
docker-compose down --volumes --remove-orphans(elasticsearch/logstash data is an external volume and will not be deleted) docker-compose build --pulldocker-compose up -d
From what I read, the queue.drain: true should force the queue to drain when stopping the container, right?
Still, after the up the logstash keeps crashing/restarting:
[2019-03-08T13:36:56,607][INFO ][org.logstash.ackedqueue.QueueUpgrade] No PQ version file found, upgrading to PQ v2.
[2019-03-08T13:36:56,861][ERROR][org.logstash.ackedqueue.QueueUpgrade] Logstash was unable to upgrade your persistent queue data.Please either downgrade to your previous version of Logstash and fully drain your persistent queue or delete your queue data.dir if you don't need to retain the data currently in your queue.
[2019-03-08T13:36:56,877][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.lang.IllegalStateException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Unexpected token (START_OBJECT), expected START_ARRAY: need JSON Array to contain As.WRAPPER_ARRAY type information for class java.util.HashMap
...
What am I missing? Is there a way to monitor the queue? Thank you in advance 