# Data in elastic search doesn't match while running the same job

**URL:** <https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080>\
**Category:** Kibana\
**Created:** [April 13, 2025, 2:24pm UTC](https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080 "2025-04-13T14:24:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![yolo1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo1/32/142597_2.png) [@yolo1](https://discuss.elastic.co/u/yolo1)\
**Post date:** [April 13, 2025, 2:24pm UTC](https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080/1 "2025-04-13T14:24:33Z")

</div>

So i have a hive script which creates multiple external table to one index and stores the data there. Later on we visualise that data on ES.

While converting the code from hive to pyspark i notice that while running the same pyspark job the data is not matching. not for all values but some values.

What could be the reason for it? I can guess that since in hive we have multiple tables so data is not mixed whereas in pyspark i am sending all datraframes t the same index. BUt if i run the same pyspark job twice i get data mismatch.  
At first i thought my code was bad but the spark and hive code return same values if i run them manually in terminal.

---

<div class="post-metadata">

**Author:** ![jessgarson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jessgarson/32/129841_2.png) [@jessgarson](https://discuss.elastic.co/u/jessgarson)\
**Post date:** [April 21, 2025, 7:12pm UTC](https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080/2 "2025-04-21T19:12:53Z")

</div>

Thanks for reaching out here. A few follow-up questions here:

- What version of Elastic are you using?
- Do you have a code example you can share with us?
- I'd also like to learn more about exactly what isn't matching.

Best,

Jessica

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [April 21, 2025, 7:27pm UTC](https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080/3 "2025-04-21T19:27:24Z")

</div>

this is duplicate post of

[https://discuss.elastic.co/t/data-mismatches-happening-while-sending-data-to-elastic-search-index-using-pyspark/377188/4](https://discuss.elastic.co/t/data-mismatches-happening-while-sending-data-to-elastic-search-index-using-pyspark/377188/4)

---

<div class="post-metadata">

**Author:** ![jessgarson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jessgarson/32/129841_2.png) [@jessgarson](https://discuss.elastic.co/u/jessgarson)\
**Post date:** [April 21, 2025, 7:56pm UTC](https://discuss.elastic.co/t/data-in-elastic-search-doesnt-match-while-running-the-same-job/377080/4 "2025-04-21T19:56:12Z")

</div>

Thanks, @elasticforme. Let's use that post for any further conversation on that subject.