For this I wrote a multithreaded writer which reads a file, bundle n
(usually 500) documents, queue the chunks which are picked up by the
writer threads which bulk index over http in round robin over all my
Now, there's a lot of tweeking that can be done to optimize
performance, see this thread for some guidelines:
On Tue, Dec 6, 2011 at 8:12 AM, ko526so firstname.lastname@example.org wrote:
I have to index huge volume of data frequently for research purpose.
60,000,000 docs are one of my recent task for indexing. Fortunately, the
size of docs is very small, so the total size of bulk index file for 60 M
docs is only 11 G.
I used the following command for Solr to prevent memory error and high
performance. And it was good.
curl http://localhost:8080/example/update -F stream.file=/tmp/artists.xml
Is there any similar command with ES like the above?