# Logstash Persistent Queues throws exception when trying to read BigInteger values from the queue

**URL:** <https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566>\
**Category:** Logstash\
**Created:** [September 23, 2017, 1:21pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566 "2017-09-23T13:21:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![subhasdan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subhasdan/32/19690_2.png) [@subhasdan](https://discuss.elastic.co/u/subhasdan)\
**Post date:** [September 23, 2017, 1:21pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/1 "2017-09-23T13:21:38Z")

</div>

- Logstash version: any version with PQs enabled - currently tested on 5.5 upto 5.6.1
- Platform: any
- Simple test case: [https://github.com/rdsubhas/logstash-queue-corruption](https://github.com/rdsubhas/logstash-queue-corruption)

logstash.yml:

```
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
xpack.monitoring.enabled: false
queue.type: persisted 

```

pipeline.conf

```
input {
  http { }
}
output {
  stdout { codec => rubydebug }
}

```

Send any event with a large int value:

```
curl -XPOST -H 'content-type:application/json'` \
  -d '{ "some_value": 9223372036854776000 }' \
  http://127.0.0.1:8080

```

And that's all. Its **full data loss from this point onwards**. Logstash will queue everything and the queue is irrecoverable.

So any event, containing any field at any nested level, from any input (http, beats, log4j2, anything) that has one overflowing int or decimal, will destroy all your logs going forward when PQs are enabled.

List of resources:

- Reproduce: [https://github.com/rdsubhas/logstash-queue-corruption](https://github.com/rdsubhas/logstash-queue-corruption)
- [Logstash 5.6.0 deserialization errors when persistent queues enabled](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952)
- [Logstash Crash with Persistent Queue and Kafka Input](https://discuss.elastic.co/t/logstash-crash-with-persistent-queue-and-kafka-input/90479/3)
- [Logstash 5.4 - Repeated deserialization errors and losing data with Persistent Queues](https://discuss.elastic.co/t/logstash-5-4-repeated-deserialization-errors-and-losing-data-with-persistent-queues/91503/2)
- PR that _might_ be a fix, scheduled for Logstash v6.1: [https://github.com/elastic/logstash/issues/8131](https://github.com/elastic/logstash/issues/8131)

### Conclusion: Logstash Persistent Queues should not be marked production ready

Please, given that any small event with a single field can cause full data loss. It's not really a small or isolated thing, the conditions are huge, varied and diverse. The magnitude of this doesn't seem to be trivial.

cc @guyboertje @Zt_Zeng @Myles

---

<div class="post-metadata">

**Author:** ![subhasdan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/subhasdan/32/19690_2.png) [@subhasdan](https://discuss.elastic.co/u/subhasdan)\
**Post date:** [September 23, 2017, 1:23pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/2 "2017-09-23T13:23:37Z")

</div>

We went with Logstash PQs as its marked ready for production use - in architecture diagrams, blog posts and everywhere, and **we trust all those announcements**. But we have experienced lots of random plugin-independent general data loss, and there is no easy solution to this, atleast until Logstash v6.1

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 23, 2017, 10:07pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/3 "2017-09-23T22:07:48Z")

</div>

Thanks for raising this, did you also create a github issue?

---

<div class="post-metadata">

**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\
**Post date:** [September 24, 2017, 1:13am UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/4 "2017-09-24T01:13:45Z")

</div>

For those following this thread, we're replying on the github issue: [https://github.com/elastic/logstash/issues/8379](https://github.com/elastic/logstash/issues/8379)

We take data-loss bugs seriously, and will plan out a course of action in the GH issue.

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [September 25, 2017, 10:27pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/5 "2017-09-25T22:27:36Z")

</div>

Followup: the problem as reported here is caused by a deserialization issue with the version of the Jackson library we are using which does not correctly deserializes Bignum/BigInteger numbers.

What is happening is, with the `{ "some_value": 9223372036854776000 }` json message, the value `9223372036854776000` is serialized as a `BigInteger` using CBOR encoding in the persisted queue, which is correct, but with the Jackson library version 2.7 that we are currently using, there is a problem with the `BigInteger` deserialization, after the Event is dequeued. The data in the queue is not corrupted but logstash is unable to decode it.

We confirmed in [https://github.com/elastic/logstash/issues/8379](https://github.com/elastic/logstash/issues/8379) that upgrading Jackson solves this problem. We are currently working to update logstash 5.6 and up with an updated Jackson version.

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [September 25, 2017, 10:51pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/6 "2017-09-25T22:51:03Z")

</div>

To follow up on @colinsurprenant's comment:

- There is no data corruption
- When a fix is released, upgrading logstash will allow this data to be read.

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [September 27, 2017, 7:32pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/7 "2017-09-27T19:32:36Z")

</div>

As discussed in [https://github.com/elastic/logstash/issues/8379](https://github.com/elastic/logstash/issues/8379) this is now fixed and will be part of the 5.6.3 release.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2017, 7:33pm UTC](https://discuss.elastic.co/t/logstash-persistent-queues-throws-exception-when-trying-to-read-biginteger-values-from-the-queue/101566/8 "2017-10-25T19:33:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.