How to handle heavy insertion


(cyrilcherian) #1

Hi,

I have 4 Elastic search nodes.
Each node having like 16 GB Ram and 100 GB or Hard Disk with a good processor.
Off the 4 nodes 2 are data nodes.

The Problem:

My application requires a huge insertion rate some think like a 400,000 documents needed to be inserted in a day.
We have text based indexing on a few fields.
Apart from the heavy insertion we also do a lot of analytic query on the documents for graphs.

What is the ideal way to do bulk insertion(so as to achieve the fast insertion rate) should I doing bulk insertion(of 1000 documents) on all the nodes or on the master node.

Thanks,
Cyril


(cyrilcherian) #2

Something i missed on my previous mail.
The index which requires heavy insertions have 20 shards.


(system) #3