Some co-workers and myself have been testing out ES recently. I wanted
to ask about some observations we've made.
Test server: 2x core i5, 4 GB RAM, 5400 RPM hard disk, Mac OS X Lion
We attempted to insert 2 million documents with a name, description,
and score. The index size was about 3 gigs.
Doing a search with a query on description for two common words
(common words are in about 10% of documents) takes about 6-7 seconds
to return 500 results, ordered by score. It takes about 2 seconds to
return the top 50 results.
Given the specs of the computer running the search, this doesn't seem
terrible. But, when running a search, we notice that ElasticSearch is
using just a few % CPU time, and less than 150 MB of RAM, even though
much more is available.
That behavior makes me think the query latency is mainly time to read
from the hard disk. But, I'm curious why ES isn't trying to use more
RAM to make the query faster.
I'm wondering if this all sounds normal, and whether there's anything
we can do to optimize this particular type of search. We changed the
index mapping to store the name, description, and score. In this case,
we don't care about the total number of matches found, if that makes a