HELP! how to disable _source and only allow a couple of fields to be stored

Elastic Stack Elasticsearch
1

For performance reasons, out of 10 fields to be indexed, I only need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from being
    stored.

  2. specify mapping for the 2 fields to "store":yes

  3. use addFields("f1","f2") to include the 2 fields to be returned:

    final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).
setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full source
store = yes or no. Did I put the config file in the right spot? How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing and
the problem will be fixed in 0.19.4:
http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

2

First, if you want to make sure that your mapping has taken affect, use the
get mapping API after the index has been created, and see the mappings it
returns.

Your solution is fine, can you share (gist) your mapping file? Lets see if
its correct or not. Btw, the location should be [config
path]/mappings/[index_name]/[type].json (note the mappings instead of
mapping).

On Thu, May 17, 2012 at 12:45 AM, Helen Poon helenchenpoon@gmail.comwrote:

For performance reasons, out of 10 fields to be indexed, I only need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from being
    stored.

  2. specify mapping for the 2 fields to "store":yes

  3. use addFields("f1","f2") to include the 2 fields to be returned:

           final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).

setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full source
store = yes or no. Did I put the config file in the right spot? How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing and
the problem will be fixed in 0.19.4:
http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

3

Thanks for your reply Shay.

Here is the mapping file. Yes, I typo'ed it here. It is actually
located at [config
path]/mappings/en/product.json

Here is the mapping file mapping file · GitHub

On May 16, 6:23 pm, Shay Banon kim...@gmail.com wrote:

First, if you want to make sure that your mapping has taken affect, use the
get mapping API after the index has been created, and see the mappings it
returns.

Your solution is fine, can you share (gist) your mapping file? Lets see if
its correct or not. Btw, the location should be [config
path]/mappings/[index_name]/[type].json (note the mappings instead of
mapping).

On Thu, May 17, 2012 at 12:45 AM, Helen Poon helenchenp...@gmail.comwrote:

For performance reasons, out of 10 fields to be indexed, I only need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from being
    stored.
  2. specify mapping for the 2 fields to "store":yes
  3. use addFields("f1","f2") to include the 2 fields to be returned:
           final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).

setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full source
store = yes or no. Did I put the config file in the right spot? How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing and
the problem will be fixed in 0.19.4:
http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

4

Here is the result from get mapping API via localhost:9200/en/product/
_mapping

Disabling of "_source" is not returned from the get mapping API.

On May 17, 9:47 am, Helen Poon helenchenp...@gmail.com wrote:

Thanks for your reply Shay.

Here is the mapping file. Yes, I typo'ed it here. It is actually
located at [config
path]/mappings/en/product.json

Here is the mapping filehttps://gist.github.com/2719028

On May 16, 6:23 pm, Shay Banon kim...@gmail.com wrote:

First, if you want to make sure that your mapping has taken affect, use the
get mapping API after the index has been created, and see the mappings it
returns.

Your solution is fine, can you share (gist) your mapping file? Lets see if
its correct or not. Btw, the location should be [config
path]/mappings/[index_name]/[type].json (note the mappings instead of
mapping).

On Thu, May 17, 2012 at 12:45 AM, Helen Poon helenchenp...@gmail.comwrote:

For performance reasons, out of 10 fields to be indexed, I only need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from being
    stored.
  2. specify mapping for the 2 fields to "store":yes
  3. use addFields("f1","f2") to include the 2 fields to be returned:
           final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).

setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full source
store = yes or no. Did I put the config file in the right spot? How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing and
the problem will be fixed in 0.19.4:
http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

5

I found the problem.... a silly one and a minor bug I think.

For those who use gedit to modify your config files and you can't
explain a problem sure you had fixed with changes to the config files,
please read on.

  1. Cause of the problem:

    I used ubuntu gedit to edit the mapping file. gedit saves your
    previous version of the file into .~. In my case, I changed _source
    to be disabled from enabled. So product.json~ is old and product.json
    is new. What thrown me off was the fact that my change in the
    mapping file did not take effect. I could not explain what was
    happening although seem silly, I I removed the product.json~ just to
    see if it made any difference. Guess what, what was it. For some
    reason, product.json~ was being picked up by elasticsearch and not the
    new file product.json. I believe this is a minor bug in the code but
    has a bad impact.

  2. Work-around, simply always delete the gedit backup files or using
    VI or something else to edit elasticsearch config files.

On May 17, 9:58 am, Helen Poon helenchenp...@gmail.com wrote:

Here is the result from get mapping API via localhost:9200/en/product/
_mapping

result from ~:9200/en/product/_mapping · GitHub

Disabling of "_source" is not returned from the get mapping API.

On May 17, 9:47 am, Helen Poon helenchenp...@gmail.com wrote:

Thanks for your reply Shay.

Here is the mapping file. Yes, I typo'ed it here. It is actually
located at [config
path]/mappings/en/product.json

Here is the mapping filehttps://gist.github.com/2719028

On May 16, 6:23 pm, Shay Banon kim...@gmail.com wrote:

First, if you want to make sure that your mapping has taken affect, use the
get mapping API after the index has been created, and see the mappings it
returns.

Your solution is fine, can you share (gist) your mapping file? Lets see if
its correct or not. Btw, the location should be [config
path]/mappings/[index_name]/[type].json (note the mappings instead of
mapping).

On Thu, May 17, 2012 at 12:45 AM, Helen Poon helenchenp...@gmail.comwrote:

For performance reasons, out of 10 fields to be indexed, I only need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from being
    stored.
  2. specify mapping for the 2 fields to "store":yes
  3. use addFields("f1","f2") to include the 2 fields to be returned:
           final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).

setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full source
store = yes or no. Did I put the config file in the right spot? How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing and
the problem will be fixed in 0.19.4:
http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

6

Nasty!, yea, elasticsearch simply iterates through the files, and tries to
parse them by auto detecting the content type, so probably the ~ one was
overriding the other one...

On Thu, May 17, 2012 at 10:48 PM, Helen Poon helenchenpoon@gmail.comwrote:

I found the problem.... a silly one and a minor bug I think.

For those who use gedit to modify your config files and you can't
explain a problem sure you had fixed with changes to the config files,
please read on.

  1. Cause of the problem:

I used ubuntu gedit to edit the mapping file. gedit saves your
previous version of the file into .~. In my case, I changed _source
to be disabled from enabled. So product.json~ is old and product.json
is new. What thrown me off was the fact that my change in the
mapping file did not take effect. I could not explain what was
happening although seem silly, I I removed the product.json~ just to
see if it made any difference. Guess what, what was it. For some
reason, product.json~ was being picked up by elasticsearch and not the
new file product.json. I believe this is a minor bug in the code but
has a bad impact.

  1. Work-around, simply always delete the gedit backup files or using
    VI or something else to edit elasticsearch config files.

On May 17, 9:58 am, Helen Poon helenchenp...@gmail.com wrote:

Here is the result from get mapping API via localhost:9200/en/product/
_mapping

result from ~:9200/en/product/_mapping · GitHub

Disabling of "_source" is not returned from the get mapping API.

On May 17, 9:47 am, Helen Poon helenchenp...@gmail.com wrote:

Thanks for your reply Shay.

Here is the mapping file. Yes, I typo'ed it here. It is actually
located at [config
path]/mappings/en/product.json

Here is the mapping filehttps://gist.github.com/2719028

On May 16, 6:23 pm, Shay Banon kim...@gmail.com wrote:

First, if you want to make sure that your mapping has taken affect,
use the
get mapping API after the index has been created, and see the
mappings it
returns.

Your solution is fine, can you share (gist) your mapping file? Lets
see if
its correct or not. Btw, the location should be [config
path]/mappings/[index_name]/[type].json (note the mappings instead of
mapping).

On Thu, May 17, 2012 at 12:45 AM, Helen Poon <
helenchenp...@gmail.com>wrote:

For performance reasons, out of 10 fields to be indexed, I only
need
to store 2 fields which are to be returned in a searchHit.

My approach in general is to:

  1. _source : {enable:false} this should disable _source from
    being
    stored.
  2. specify mapping for the 2 fields to "store":yes
  3. use addFields("f1","f2") to include the 2 fields to be
    returned:
           final SearchResponse response =

client.prepareSearch(getIndexName()).setTypes(getIndexType()).

setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setQuery(query.getQuery()).
setFilter(query.getFilter()).
setFrom(startAt).
setSize(maxHits).
setExplain(true).
addFields("f1", "f2").
execute().actionGet();

  1. From searchHit, use eshit.getFields().get("f1").getValue() to
    get
    the returned search hit field value.

Question 1: Is this the right approach?

Question 2: How to specify "_source : {enable:false} "? I
currently
not able to make this work. I am using 0.19.3 and I have specified
this in a mapping file located at /config/mapping/[index
name]/[index
type].json. I see the _source when using GET and SEARCH API. I am
using Java client, the searchHit.getSource() returns the full
source
store = yes or no. Did I put the config file in the right spot?
How
can I use Java client to disable _source?

Question 3: Please confirm the following is what I am experiencing
and
the problem will be fixed in 0.19.4:

http://groups.google.com/group/elasticsearch/browse_frm/thread/10e54c...

I am a newbie here but I do have lucene experience. I really
appreciate your help here.

BTW, are there any books on elasticsearch?

7

Shay,

I retried the same setup mutliple times and then restarted the elastic search.
However, the mapping file is not picked up.

This is what i did.

  1. created a mapping file "datastore.json" with the following configuration

{"datastore":{"_source":{"enabled":false},"properties":{"data1":{"type":"string","store":"yes","index":"not_analyzed"}}}}.

  1. Copied under /config/mappings/es/datastore.json
  2. Deleted the existing mappings under the type 'datastore'.
  3. Restarted the elasticsearch.

Kindly verify.