Elastic search fails to index the records once it reaches 10000

Elastic Stack Elasticsearch
1

Hello,

I'm new to elastic search so pardon me if you think this question is silly. So i'm facing issues while adding the records to my index. As soon as 10000 records are added to the index, my index doesn't take the records beyond that. Like for example if i'm trying to insert 20000 records at once, only 10000 records gets indexed. No documents are indexed after 10000 records.
I tried increasing the heap size still no improvement. This issue persist not only for single index but also for the multiple index having records above 10000.
Please help.
Thanks in advance

3

Hi,
How are you querying your index to determine how many items it holds?
Could it be that you are running into the top hits feature of ES7?
https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#request-body-search-track-total-hits
Derk

4

Hi Derk,

Thanks for the reply.
yes you are right i was passing the wrong query to see the number of items in the index.
I used _count to check the total number of items in the index and its showing right.
Thank you so much.. Also i wanted to fetch all the items inside my index but currently only 10000 records is returned. Any workaround to fetch all the records from index?

Thanks

5

Hi Nimisha,

If you want to loop through a complete resultset, you should use the scroll api.
https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search-request-body.html#request-body-search-scroll

Derk

6

I have used scroll to fetch the records..But my motive is to retrieve 6-7Lakhs of data on one go. In my current scenario i'm fetching 5k data at a time using scroll with while loop. I actually want to overcome the loop so as to avoid overhead.

7

Hi Nimisha,
Ok, I get your point, although I still believe that the scroll api is the way to go.
If you really wanted to, you could override the index.max_result_window setting, although I don't believe that is recommended, since it would eat into the heap memory and might cause OOM exceptions at some point.
Derk

8

Thanks derk it helped..Thank you so much :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.