Migrating from 5.6 to 6.6 and issue with persistent queue

Hi there,

we are using a very simple one node ELK docker-compose based stack running on one server. Elasticsearch, Kibana, Logstash are all running version 5.6.3.

We use the persistent queue feature. I'm aware of the logstash issue with persistent queues and I think I made the required change already in the past.

Both the "old" (and restarted/running) Logstash 5.6 and new 6.6 config/logstash.yml (queue related content):

queue.type: persisted
queue.drain: true
path.queue: /usr/share/logstash/queue

What we do:

  1. copy 6.6 files to target directory
  2. docker-compose down --volumes --remove-orphans (elasticsearch/logstash data is an external volume and will not be deleted)
  3. docker-compose build --pull
  4. docker-compose up -d

From what I read, the queue.drain: true should force the queue to drain when stopping the container, right?

Still, after the up the logstash keeps crashing/restarting:

[2019-03-08T13:36:56,607][INFO ][org.logstash.ackedqueue.QueueUpgrade] No PQ version file found, upgrading to PQ v2.
[2019-03-08T13:36:56,861][ERROR][org.logstash.ackedqueue.QueueUpgrade] Logstash was unable to upgrade your persistent queue data.Please either downgrade to your previous version of Logstash and fully drain your persistent queue or delete your queue data.dir if you don't need to retain the data currently in your queue.
[2019-03-08T13:36:56,877][ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.
java.lang.IllegalStateException: com.fasterxml.jackson.databind.exc.MismatchedInputException: Unexpected token (START_OBJECT), expected START_ARRAY: need JSON Array to contain As.WRAPPER_ARRAY type information for class java.util.HashMap

What am I missing? Is there a way to monitor the queue? Thank you in advance :slight_smile:

Looks like I finally found the issue. The queue.drain setting can be found first in 6.0 logstash documentation . (No word about this setting in 5.6). So it looks like I have to add at least one more step (5.6. -> 6.0 -> 6.6)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.