I'm currently evaluating ElasticSearch as a replacement for our
existing Solr search indices and initial tests have proved very
promising. However, our largest index is just shy of 30m documents and
naturally weighs in at quite a hefty size. Indexing this in ES using
the standard index storage type (mmapfs, I think) and a 10gb heap
(-Xmx10g) works fine but when querying the index I get maybe 3-4
requests/s compared to Solr's ~148 requests/s. If I then shut down the
ES server and start it with "-Des.index.store.type=memory" the number
of requests shoots up by almost a factor of 100 (~280-300 requests/s).
Next I cleared out the index and tried to re-index using an in-memory
index from the start, but doing it this way I get out of memory errors
half way through indexing (the full index size is ~20gb).
Am I doing something wrong here? How can the index work fine if I
index it and then shift to in-memory, but not directly as in-memory?
Any help appreciated,