# Logstash 5.6.0 deserialization errors when persistent queues enabled

**URL:** <https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952>\
**Category:** Logstash\
**Created:** [September 19, 2017, 1:52am UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952 "2017-09-19T01:52:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zt\_Zeng](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zt\_Zeng](https://discuss.elastic.co/u/Zt_Zeng)\
**Post date:** [September 19, 2017, 1:52am UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/1 "2017-09-19T01:52:33Z")

</div>

[The same problem in a closed post](https://discuss.elastic.co/t/logstash-5-4-repeated-deserialization-errors-and-losing-data-with-persistent-queues/91503?u=zt_zeng)

- Version: 5.5.0 & 5.6.0

- OS: Linux ubuntu-100 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86\_64 x86\_64 x86\_64 GNU/Linux

- Config file: [logstash.yml](https://github.com/zzt93/log_egg/blob/master/logstash.yml) with

- Pipeline config: [bussiness.conf](https://github.com/zzt93/log_egg/blob/master/bussiness.conf)

- Sample Data:

- Steps to reproduce:  
Just run with above data and config, If we enable 'persistent queue', it will throw the error. Otherwise, it works.

---

<div class="post-metadata">

**Author:** ![subhasdan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subhasdan/32/19690_2.png) [@subhasdan](https://discuss.elastic.co/u/subhasdan)\
**Post date:** [September 19, 2017, 3:07pm UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/2 "2017-09-19T15:07:00Z")

</div>

Thank you so much for clearly reproducing!! This issue was driving us crazy and losing lot of logs on random days!

@guyboertje can you please share your thoughts on this 🙂

---

<div class="post-metadata">

**Author:** ![subhasdan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subhasdan/32/19690_2.png) [@subhasdan](https://discuss.elastic.co/u/subhasdan)\
**Post date:** [September 19, 2017, 3:47pm UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/3 "2017-09-19T15:47:53Z")

</div>

@Zt_Zeng also - after your receive this error, does the pipeline continue to keep working/flowing? Because for us the deserialization error gets stuck there, just want to confirm if you see the same behavior?

(and btw - the JSON has an extra [0] in the message, so semantically not valid JSON, just to make sure if its the exact same message produced by filebeat)

---

<div class="post-metadata">

**Author:** ![subhasdan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subhasdan/32/19690_2.png) [@subhasdan](https://discuss.elastic.co/u/subhasdan)\
**Post date:** [September 22, 2017, 8:28pm UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/4 "2017-09-22T20:28:15Z")

</div>

@Zt_Zeng how can we test the message by sending it directly to beat input 5044?

---

<div class="post-metadata">

**Author:** ![Zt\_Zeng](https://avatars.discourse-cdn.com/v4/letter/z/b77776/32.png) [@Zt\_Zeng](https://discuss.elastic.co/u/Zt_Zeng)\
**Post date:** [September 23, 2017, 1:27am UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/5 "2017-09-23T01:27:16Z")

</div>

I think it isn't working, because no data is processed from the log and ES not receive any data.

As for the strange `[0]`, I reproduce the problem, and it is the exact message received by **logstash**. But this is the log printed when no persistent queue is enabled, because when logstash start with persistent queue, I can only see error stack trace and no received event.

And the following is the json published by filebeat:

```
{
  "@timestamp": "2017-09-23T01:19:01.448Z",
  "beat": {
    "hostname": "server",
    "name": "server",
    "version": "5.5.2"
  },
  "fields": {
    "type": "biz-info"
  },
  "input_type": "log",
  "message": "xxx",
  "offset": 268,
  "source": "xxx/log_info.2017-09-21.log",
  "type": "log"
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 21, 2017, 1:28am UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952/6 "2017-10-21T01:28:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.