# Logstash5.5 to kafka newer version - data discripency

**URL:** <https://discuss.elastic.co/t/logstash5-5-to-kafka-newer-version-data-discripency/233257>\
**Category:** Logstash\
**Created:** [May 19, 2020, 6:58am UTC](https://discuss.elastic.co/t/logstash5-5-to-kafka-newer-version-data-discripency/233257 "2020-05-19T06:58:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![manu1](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@manu1](https://discuss.elastic.co/u/manu1)\
**Post date:** [May 19, 2020, 6:58am UTC](https://discuss.elastic.co/t/logstash5-5-to-kafka-newer-version-data-discripency/233257/1 "2020-05-19T06:58:53Z")

</div>

Hello,

We are using Logstash5.5 send event logs to Kafka newer version(2.4). We are seeing some data discrepancy. Issue seems to be resolving if we restart logstash service, however it is reoccurring nearly in 48 hours every time and we ended up restarting as we dont have root cause of this issue.

We have captured node api stats and created dashboards in grafana.  
We couldn't see any metrics which is alarming.

```auto
    - wait for worker % < 1% ==> ((queue_push_duration_in_millis/duration_in_millis)*100)
    - Queue(persistent)% used < 3% 
    - Heap % used < 30%

```

Also, enabled DEBUG logs for logstash and couldn't find any specific errors or failures. Kafka cluster is healthy and all other pipelines which is not using logstash5 is good.

We understand logstash5.5 used _Kafka client version 0.10_, our understanding it is compatible with higher version of kafka also.

- Does it cause of any data discripency?
- Any performance issue or compatibility issue for using logstash5.5 --\> Kafka 2.4?
- Our legacy pipeline was using logstash1.5 --\> Kafka 0.8, which is still working fine.

Appreciate any inputs on this issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 16, 2020, 6:58am UTC](https://discuss.elastic.co/t/logstash5-5-to-kafka-newer-version-data-discripency/233257/2 "2020-06-16T06:58:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.