Logs hit value is different

[Aniket_Pant (kaal)]

I have two elasticsearch cluster one elk cluster version is 7.6 and other elk cluster version is 7.11
The former elasticsearch cluster is simple it filebeat send the logs to logstash and logstash to elasticsearch, the latter is different we have kafka before the logstash we are not using filebeat here...the source send logs to kafka and logstash consumes the logs from kafka and it send logs to elasticsearch
The pipeline file(filter+output) is same
I don't know why logs hit value is different? so is there a difference taking logs from filebeat and taking logs from kafka
ELK-7.6

ELK-7.11

[Miguel1 (Miguel)]

From some versions until now, hit count or cardinality queries are not exact in order to improve query times. Here you can find some more information.

[Aniket_Pant (kaal)]

Ok i understand your point but is there any difference between Kafka and filebeat like which effect logs hits value

[Miguel1 (Miguel)]

If your Kafka is properly configured and all the logstash consumers have access to the same offset (consumer group), it should be the same. In addition, Logstash and Filebeat ensure at- least - once delivery, so the messages can be delivered more than one time. Anyway, the gap between the two samples is too big. In order to discard any problem with Elasticsearch you could use file output on both and wc -l the lines. I would check Kafka conf and Elasticsearch mappings, maybe one index is getting refreshed more than the other.

[Aniket_Pant (kaal)]

Sorry i don't get it i have to use file output for testing

[Aniket_Pant (kaal)]

@Miguel1

would check Kafka conf and Elasticsearch mappings, maybe one index is getting refreshed more than the other.

I have to paste kafka conf file here and elasticsearch mapping also.

[leandrojmp (Leandro Pereira)]

What is the source? Could be that it is sending less logs to Kafka for some reason?

Try to find some log that exists when shipped by filebeat and do not exists consumed from Kafka by logstash.

After that consume from your kafka using another tool, the console consumer tool for example, to see if the logs is there.

You could use a small time interval in Kibana, something like 1 minute or 30 seconds, to create a report in both cases and compare it to find a message that exists when sent by filebeat and does not exists when consumed by logstash from Kafka.

[Aniket_Pant (kaal)]

The source is sending same number of logs to kafka and filebeat i got the issue but not so proper
when the ship from filebeat to logstash and then to elasticsearch the log count is same

[root@stag kafka]# wc -l /var/log/vpn1/2021.06.13logs.log
3275 /var/log/vpn1/2021.06.13logs.log

logs ship though filebeat(source->filebeat->logstash->elasticsearch)

Case 2:
when we use kafka before sending it to logstash i ran bin/kafka-console-consumer.sh command to consume logs the log count was 3275
^CProcessed a total of 3275 messages
but when it reached to elasticsearch the count is different although pipeline file is same

I have 2 different ELK but i can't find where it is all happening pipeline file is same