Regarding the crashes, both Lucene itself and elasticsearch go through
great effort to not corrupt the data in case of out of memory or other
problems, like open file handles. I actually have several (non automated
tests) that I run regularly that simulate those problems with no data loss.
If you can help in trying to recreate what you saw, it would be great!
Regarding recovery of data, I have been thinking, and mentioned on the
mailing list, of the ability to snapshot an index to a shared storage when
using local gateway (basically, combine the shared gateway snapshot
capabilities with local gateway). The main thing to note with this is the
fact that it takes time to transfer large amount of data back to the nodes
in case a full recovery is needed...
On Thu, Aug 4, 2011 at 2:21 AM, Tom Le email@example.com wrote:
I am up to 200M documents inserted per day using HTTP REST API bulk
insertion with refresh settings from 5s to 20s.
The only problems I've had so far:
- Transaction logs take up a lot of space, which I believe 0.17.3 will
- Some operational issues had catastrophic impact, such as not having
enough open files and the index getting corrupted.
- Smaller java heap size that ran out of memory, when crashing, sometimes
corrupted the index. I tried to reproduce but corruption on crash doesn't
occur in all cases..
- Index and search response can be slow for a few minutes when starting up
ES and you have really large indexes.
I would like a way to recover an index short of re-indexing, as 6 months
down the road, I do not want to re-index everything should something
On Wed, Aug 3, 2011 at 1:58 PM, Jay Taylor firstname.lastname@example.org wrote:
I've been ingesting a few hundred GB per day now with ES without any major