Kibana not showing all data

Elastic Stack Kibana
#1

Hi guys , recently i injected a 1,7 GB file in elasticsearch using logstash, so when i check the index management i find 9782102

image
image862×45 2.83 KB

and when i try to make a metric visualization it gives me 9007610 document
image
image1848×618 29.9 KB

any idea whats the problem, is it a performance problem ? i have 8Gb of RAM and 4GB heap size!!

ELK 7.4.0.

Thanks in advance.

#2

What version of Kibana and ES?

Does the total Count on the discover tab (top right corner) match the count you get when hitting Elasticsearch directly? If not, try opening developer tools in your browser and look at the requests Kibana is sending to elasticsearch. On the Discover tab you should see a couple of msearch requests. Are they querying the indexes you'd expect?

Also some info mentioned in this thread might be of use: Kibana not showing recent Elasticsearch data

Thanks
Rashmi

#3

thanks for the reply , i'm using ELK 7.4.0 and the discover tab shows the same number as the index management tab
image

#4

@flash1293 - what might be the issue here?

How do you count the number of ES documents? Are you e.g. including the documents in the .kibana index, which Kibana won't show? Can you narrow down the different to a particular time interval, i.e. what if you pick a random month (or day) and compare the counts in Kibana and directly in ES?

#5

I'm not including the .kibana index , i checked the index management tab which gives the number of documents and the discover tab also gives the same number , but when i try to create a visualization that displays the number of documents it gives a whole different number!!

#6

what happens when you hit ES to get the exact count ? Are there more instances which you are running ?

#7

hi , it works now , recently i changed "search.max_buckets" which caused the problem i guess !! because when i changed back to 10000 , the visualization gave the same count as the index management tab and the discover tab , but i'm facing another issue in the map visualization , when i make my filter last 1 year the sum aggregation in the map return result but when i change the filter to the last 30 days it returns 0 , though i have data in my documents (the count aggregation works) and this is only in the map visualization

#8

Thanks for getting back with the solution that worked.

Cheers
Rashmi