Specific fields cache

sxamt · September 30, 2012, 8:06am

Hi,
I want to use elastic search the following way:
On start up I want the Elastic Search to load all "my id fields" (one or
two small fields per document connecting between Lucene Doc Id and my
entities). The reason for that is I don't want to get Doc Id as a search
results and again make elastic search go to the disk to fetch me the "my id
fields" for documents found. Currently this is the way I work with Lucene
directly and I am pretty happy about it. Is there a way I can make elastic
search work the same way, in order to avoid another Lucene Db trip?

Thanks!

Maxim

--

Ivan · October 1, 2012, 5:44pm

Elasticsearch 0.20.0 will have a warmup API that might help:

github.com/elastic/elasticsearch

Index Warmup API

opened 03:49PM - 06 May 12 UTC

closed 03:50PM - 06 May 12 UTC

kimchy

>feature v0.20.0.RC1

Index warming allows to run registered search requests to warm up the index befo…re it is available for search. With the near real time aspect of search, cold data (segments) will be warmed up before they become available for search. Warmup searches typically include requests that require heavy loading of data, such as faceting or sorting on specific fields. The warmup APIs allows to register warmup (search) under specific names, remove them, and get them. Index warmup can be disabled by setting `index.warmer.enabled` to `false`. It is supported as a realtime setting using update settings API. This can be handy when doing initial bulk indexing, disabling pre registered warmers to make indexing faster and less expensive and then enable it. ## Put Warmer Allows to put a warmup search request on a specific index (or indices), with the body composing of a regular search request. Types can be provided as part of the URI if the search request is designed to be run only against the specific types. Here is an example that registers a warmup called `warmer_1` against index `test` (can be alias or several indices), for a search request that runs against all types: ``` curl -XPUT localhost:9200/test/_warmer/warmer_1 -d '{ "query" : { "match_all" : {} }, "facets" : { "facet_1" : { "terms" : { "field" : "field" } } } }' ``` And an example that registers a warmup against specific types: ``` curl -XPUT localhost:9200/test/type1/_warmer/warmer_1 -d '{ "query" : { "match_all" : {} }, "facets" : { "facet_1" : { "terms" : { "field" : "field" } } } }' ``` ## Delete Warmer Removing a warmer can be done against an index (or alias / indices) based on its name. The provided name can be a simple wildcard expression or omitted to remove all warmers. Some samples: ``` # delete warmer named warmer_1 on test index curl -XDELETE localhost:9200/test/_warmer/warmer_1 # delete all warmers that start with warm on test index curl -XDELETE localhost:9200/test/_warmer/warm* # delete all warmers for test index curl -XDELETE localhost:9200/test/_warmer/ ``` ## GETting Warmer Getting a warmer for specific index (or alias, or several indices) based on its name. The provided name can be a simple wildcard expression or omitted to get all warmers. Some examples: ``` # get warmer named warmer_1 on test index curl -XGET localhost:9200/test/_warmer/warmer_1 # get all warmers that start with warm on test index curl -XGET localhost:9200/test/_warmer/warm* # get all warmers for test index curl -XGET localhost:9200/test/_warmer/ ```

The issue does not go into much detail besides that it works with new
segments, but I am assuming it will populate the field caches as well.

If you have source enabled, Elasticsearch should retrieve the entire
resulting document in one call instead of a separate call for each field.
Not sure if source is retrieve during search however or if is still another
Lucene DB trip.

Cheers,

Ivan

On Sun, Sep 30, 2012 at 1:06 AM, Maxim Terletsky sxamt33@gmail.com wrote:

Hi,
I want to use Elasticsearch the following way:
On start up I want the Elastic Search to load all "my id fields" (one or
two small fields per document connecting between Lucene Doc Id and my
entities). The reason for that is I don't want to get Doc Id as a search
results and again make Elasticsearch go to the disk to fetch me the "my id
fields" for documents found. Currently this is the way I work with Lucene
directly and I am pretty happy about it. Is there a way I can make elastic
search work the same way, in order to avoid another Lucene Db trip?

Thanks!

Maxim

--

--