I'm experiencing problems when using the bulk feature of Elastic but I'm still very novice in ES so I don't know how to even debug it properly.
I'm trying to index 2 million documents using the bulk feature with each chunk being 1000 documents. Everything is going good in the first 100k documents, 100k - 150k is kinda slow, 150k - 200k is very slow and 200k+ document is extremely slow. To give you a perspective on 200k+ ES is indexing 1000 documents every 5 mins.
- elasticsearch: 7.4.2,
- Windows: 10,
- Ram: 16 GB,
- Processor: intel i-7 8550U
I haven't touched any of the es configuration files.
What I tried:
Everything was based on something i read from the web and nothing really helped. When i create the index I tried using these settings:
[ 'number_of_shards' => 5, 'number_of_replicas' => 1, 'refresh_interval' => '1s' ],
these are my default settings. I tried also with the following with no success:
[ 'number_of_shards' => 5, 'number_of_replicas' => 0, 'refresh_interval' => '30s' ]
I would appreciate any help/suggestion. I will supply more info if needed.
Thank you for your time.