# Logs hit value is different **URL:** \ **Category:** Logstash\ **Created:** [June 11, 2021, 10:34am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675 "2021-06-11T10:34:06Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [June 11, 2021, 10:34am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/1 "2021-06-11T10:34:06Z")
I have two elasticsearch cluster one elk cluster version is 7.6 and other elk cluster version is 7.11 The former elasticsearch cluster is simple it filebeat send the logs to logstash and logstash to elasticsearch, the latter is different we have kafka before the logstash we are not using filebeat here...the source send logs to kafka and logstash consumes the logs from kafka and it send logs to elasticsearch The pipeline file(filter+output) is same I don't know why logs hit value is different? so is there a difference taking logs from filebeat and taking logs from kafka **ELK-7.6** ![elk7.6](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a698d1a3e0cdda6aacca60b2fc7626cb44514baa.jpeg) **ELK-7.11** ![elk7.11](https://us1.discourse-cdn.com/elastic/original/3X/4/7/47bc3670547710ed2b785d18df4efa09b522aec6.jpeg) ---
**Author:** ![Miguel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel1/32/82094_2.png) [@Miguel1](https://discuss.elastic.co/u/Miguel1)\ **Post date:** [June 11, 2021, 10:39am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/2 "2021-06-11T10:39:17Z")
From some versions until now, hit count or cardinality queries are not exact in order to improve query times. [Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html) you can find some more information. ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [June 11, 2021, 11:04am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/3 "2021-06-11T11:04:00Z")
Ok i understand your point but is there any difference between Kafka and filebeat like which effect logs hits value ---
**Author:** ![Miguel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel1/32/82094_2.png) [@Miguel1](https://discuss.elastic.co/u/Miguel1)\ **Post date:** [June 11, 2021, 1:40pm UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/4 "2021-06-11T13:40:55Z")
If your Kafka is properly configured and all the logstash consumers have access to the same offset (consumer group), it should be the same. In addition, Logstash and Filebeat ensure at- **least** - **once** delivery, so the messages can be delivered more than one time. Anyway, the gap between the two samples is too big. In order to discard any problem with Elasticsearch you could use file output on both and **wc -l** the lines. I would check Kafka conf and Elasticsearch mappings, maybe one index is getting refreshed more than the other. ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [June 11, 2021, 2:41pm UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/5 "2021-06-11T14:41:25Z")
Sorry i don't get it 😅 i have to use file output for testing ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [June 11, 2021, 2:42pm UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/6 "2021-06-11T14:42:36Z")
> [@Miguel1](#): > > would check Kafka conf and Elasticsearch mappings, maybe one index is getting refreshed more than the other. I have to paste kafka conf file here and elasticsearch mapping also. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 11, 2021, 3:12pm UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/7 "2021-06-11T15:12:47Z")
What is the source? Could be that it is sending less logs to Kafka for some reason? Try to find some log that exists when shipped by filebeat and do not exists consumed from Kafka by logstash. After that consume from your kafka using another tool, the console consumer tool for example, to see if the logs is there. You could use a small time interval in Kibana, something like 1 minute or 30 seconds, to create a report in both cases and compare it to find a message that exists when sent by filebeat and does not exists when consumed by logstash from Kafka. ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [June 13, 2021, 10:40am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/8 "2021-06-13T10:40:30Z")
The source is sending same number of logs to kafka and filebeat i got the issue but not so proper when the ship from filebeat to logstash and then to elasticsearch the log count is same ```auto [root@stag kafka]# wc -l /var/log/vpn1/2021.06.13logs.log 3275 /var/log/vpn1/2021.06.13logs.log ``` **logs ship though filebeat(source-\>filebeat-\>logstash-\>elasticsearch)** ![vpn1](https://us1.discourse-cdn.com/elastic/original/3X/d/1/d179f97ad29a8126e72280179278c1bc18a870ef.jpeg) **Case 2:** when we use kafka before sending it to logstash i ran `bin/kafka-console-consumer.sh` command to consume logs the log count was 3275 ` ^CProcessed a total of 3275 messages` but when it reached to elasticsearch the count is different although pipeline file is same ![vpn11](https://us1.discourse-cdn.com/elastic/original/3X/c/a/ca1f6896d59a103921c16ad8a9dff32c0ccf8fe7.jpeg) I have 2 different ELK but i can't find where it is all happening pipeline file is same ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 11, 2021, 10:41am UTC](https://discuss.elastic.co/t/logs-hit-value-is-different/275675/9 "2021-07-11T10:41:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.