We were stoked when we found out about the updating feature in the recent
0.19.0rc2 release. We have been eagerly experimenting with it but are
disappointed by it's performance. Hopefully you can tell us we are doing
We roughly use this model: https://gist.github.com/1751349. Starting from a
clean index it takes 7 seconds to index 1000 documents (ok-ish). After
indexing 3 million documents performance degrades to 30 seconds per 1000
documents (prohibitively slow). We expect to insert 500 million documents
plus 4 million a day.
Our approach inserting documents is as follows:
We first try to update a document, if that returns an error we instead
The resulting documents can contain hundreds and possibly thousands of
'interactions' growing the document size to about 3Mb.
Are there ways of speeding this process up?
With kind regards,