How to hide some fields in _store using mappings


(akram79) #1

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #2

On Wednesday, November 20, 2013 10:09:20 PM UTC+5:30, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #3

Elasticsearch does not modify the source, so you cannot change any part of
it when it is stored. You can store individual fields and retrieve their
value given the document. By default, elasticsearch does not store fields,
which the requires having the source to retrieve the field values. So
basically you can disable the source and enable storing only the fields you
want. Look at the store option:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

Keep in mind that you would need to retrieve each field individually, which
means one seek per field, as opposed to one seek for the original document
source. For a few fields, the difference is negligible, but might be an
issue if you are using lots of stored fields.

Cheers,

Ivan

On Wed, Nov 20, 2013 at 8:39 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #4

Hi Ivan,

Thanks for reply.

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Regards,
Akram.

On Wednesday, November 20, 2013 10:28:54 PM UTC+5:30, Ivan Brusic wrote:

Elasticsearch does not modify the source, so you cannot change any part of
it when it is stored. You can store individual fields and retrieve their
value given the document. By default, elasticsearch does not store fields,
which the requires having the source to retrieve the field values. So
basically you can disable the source and enable storing only the fields you
want. Look at the store option:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

Cheers,

Ivan

On Wed, Nov 20, 2013 at 8:39 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #5

Hello Akram,

In the future, please respond to the mailing list and not back to whoever
responded to you since there are others that can help and perhaps others in
the future with the same issue can read the replies. Also please note that
you sent three emails to the list and then two directly to me in a very
short time span. This action is incredibly rude. Please do not spam.

After retrieve the SearchHits from the SearchResponse object, you can use
the various field methods:
http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/search.html
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/search/SearchHit.java?source=cc#L142-L154

To add fields to an existing document, you can either reindex or use the
update API:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html

There is no Java API documentation, but the best way to explore how the
Java API works is via the unit tests. Here are some tests for the update
API:
https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/update/UpdateTests.java

Cheers,

Ivan

On Wed, Nov 20, 2013 at 9:11 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi Ivan,

Thanks for reply.

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Regards,
Akram.

On Wednesday, November 20, 2013 10:28:54 PM UTC+5:30, Ivan Brusic wrote:

Elasticsearch does not modify the source, so you cannot change any part
of it when it is stored. You can store individual fields and retrieve
their value given the document. By default, elasticsearch does not store
fields, which the requires having the source to retrieve the field values.
So basically you can disable the source and enable storing only the fields
you want. Look at the store option:

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/mapping-core-types.html

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

Cheers,

Ivan

On Wed, Nov 20, 2013 at 8:39 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #6

Hi Ivan,

I was not properly aware of this UI, so I followed usual, gmail with reply
to Author button.

Sorry for Inconvenience.


Still I have some doubts on query posted.

When I set _source : {"enabled" : false}, I am not able to get value of
fields, If I try to get each field individually by
searchHit.getField("fieldName"). As I looked into the method getField it
internally fetching from source map only, where it is null.

Do I have any other options to get the field values.

Regards,
Akram.

On Wednesday, November 20, 2013 11:27:48 PM UTC+5:30, Ivan Brusic wrote:

Hello Akram,

In the future, please respond to the mailing list and not back to whoever
responded to you since there are others that can help and perhaps others in
the future with the same issue can read the replies. Also please note that
you sent three emails to the list and then two directly to me in a very
short time span. This action is incredibly rude. Please do not spam.

After retrieve the SearchHits from the SearchResponse object, you can use
the various field methods:

http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/search.html

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/search/SearchHit.java?source=cc#L142-L154

To add fields to an existing document, you can either reindex or use the
update API:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html

There is no Java API documentation, but the best way to explore how the
Java API works is via the unit tests. Here are some tests for the update
API:

https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/update/UpdateTests.java

Cheers,

Ivan

On Wed, Nov 20, 2013 at 9:11 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi Ivan,

Thanks for reply.

Can you let me know, how to retrieve each field using JAVA API and also I
need to add some of the fields at runtime to _source using java API. If
possible can you please let me know, how to achieve it.

Regards,
Akram.

On Wednesday, November 20, 2013 10:28:54 PM UTC+5:30, Ivan Brusic wrote:

Elasticsearch does not modify the source, so you cannot change any part
of it when it is stored. You can store individual fields and retrieve
their value given the document. By default, elasticsearch does not store
fields, which the requires having the source to retrieve the field values.
So basically you can disable the source and enable storing only the fields
you want. Look at the store option:

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/mapping-core-types.html

Keep in mind that you would need to retrieve each field individually,
which means one seek per field, as opposed to one seek for the original
document source. For a few fields, the difference is negligible, but might
be an issue if you are using lots of stored fields.

Cheers,

Ivan

On Wed, Nov 20, 2013 at 8:39 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Hannes Korte) #7

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #8

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the source
from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the correct
    data, with remaining fields displayed not displaying "data" field - it is
    fine
    .
  3. To the ssme document added some other fields like "assigned", etc - They
    were added to document.
  4. But when I searched once again for "dataone" using java api, now the
    search is not returning the data.

how to make "data" field, should not be lost after updated - I guess it
lost but not sure, but not returning in the search result at second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #9

It would help if you provided your mapping and an example query. Are you
searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/ It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some updates
to the mapping will not affect documents that are currently indexed. How
are you adding fields to a document? Are you reindexing or using the update
API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the source
from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the correct
    data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc -
    They were added to document.
  4. But when I searched once again for "dataone" using java api, now the
    search is not returning the data.

how to make "data" field, should not be lost after updated - I guess it
lost but not sure, but not returning in the search result at second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/mapping-source-field.html

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #10

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

  • .setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*
    builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();
scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type, id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are you
searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the source
from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the correct
    data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc -
    They were added to document.
  4. But when I searched once again for "dataone" using java api, now the
    search is not returning the data.

how to make "data" field, should not be lost after updated - I guess it
lost but not sure, but not returning in the search result at second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #11

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required if
you want to search against it. It works after indexing, but will disappear
after updating. I doubt that is a bug, since elasticsearch has nothing to
update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

  • .setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is
lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are you
searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the correct
    data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc -
    They were added to document.
  4. But when I searched once again for "dataone" using java api, now the
    search is not returning the data.

how to make "data" field, should not be lost after updated - I guess it
lost but not sure, but not returning in the search result at second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #12

Hi,

yes, I have missed the line, the code is, as it was returned from other
method, I missed it.

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder = QueryBuilders.matchPhrasePrefixQuery("data",
"dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required if
you want to search against it. It works after indexing, but will disappear
after updating. I doubt that is a bug, since elasticsearch has nothing to
update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

  • .setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is
lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are you
searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc -
    They were added to document.
  4. But when I searched once again for "dataone" using java api, now the
    search is not returning the data.

how to make "data" field, should not be lost after updated - I guess it
lost but not sure, but not returning in the search result at second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned as
null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #13

I do not see the issue on 0.90.5. After an update I can no longer search
against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from other
method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(
searchContext.getIndexName()).setSearchType(SearchType.DFS_
QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder = QueryBuilders.matchPhrasePrefixQuery("data",
"dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(
FilterBuilders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

  • .setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is
lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are
you searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc -
    They were added to document.
  4. But when I searched once again for "dataone" using java api, now
    the search is not returning the data.

how to make "data" field, should not be lost after updated - I guess
it lost but not sure, but not returning in the search result at second
time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte wrote:

Hi Akram,

besides the solution of storing each field individually, you could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #14

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search first
time on the "data" field, but after updating, not able to search on the
"data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer search
against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

yes, I have missed the line, the code is, as it was returned from other
method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(
searchContext.getIndexName()).setSearchType(SearchType.DFS_
QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder = QueryBuilders.matchPhrasePrefixQuery("data",
"dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(
FilterBuilders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query! Which
field are you searching against? You mention that the data field is lost,
but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

  • .setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is
lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are
you searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" : "dataone",
    "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc
  • They were added to document.
  1. But when I searched once again for "dataone" using java api, now
    the search is not returning the data.

how to make "data" field, should not be lost after updated - I guess
it lost but not sure, but not returning in the search result at second
time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you could
as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #15

Yes, I did ask before if that was a typo. :slight_smile:

An update in Lucene is basically an atomic delete and update. Elasticsearch
needs the source if it is to preserve the content between updates. I would
argue that the real bug here is that elasticsearch should not have indexed
the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

Cheers,

Ivan

On Fri, Nov 22, 2013 at 10:19 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search first
time on the "data" field, but after updating, not able to search on the
"data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer search
against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from other
method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(searchCon
text.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder =
QueryBuilders.matchPhrasePrefixQuery("data", "dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuil
ders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field itself,
after updating/adding new fields to the source, the field "data" is getting
lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is required
if you want to search against it. It works after indexing, but will
disappear after updating. I doubt that is a bug, since elasticsearch has
nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

*
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field is
lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic wrote:

It would help if you provided your mapping and an example query. Are
you searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <akram.hu...@gmail.com

wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" :
    "dataone", "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned", etc
  • They were added to document.
  1. But when I searched once again for "dataone" using java api, now
    the search is not returning the data.

how to make "data" field, should not be lost after updated - I guess
it lost but not sure, but not returning in the search result at second
time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you could
as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part of
the
_source.

when I set _source : { enabled : false}, source is being returned
as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #16

Hi,

As the "data" field is sensitive data, customer don't want it to be visible
as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo. :slight_smile:

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

Cheers,

Ivan

On Fri, Nov 22, 2013 at 10:19 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search first
time on the "data" field, but after updating, not able to search on the
"data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer search
against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from other
method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(searchCon
text.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder =
QueryBuilders.matchPhrasePrefixQuery("data", "dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuil
ders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

*
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field
is lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic
wrote:

It would help if you provided your mapping and an example query. Are
you searching against a certain field, the data field, or the _all field?
Create a gist as described here: http://www.elasticsearch.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be searchable.

  1. With the template I have created the doc like {"data" :
    "dataone", "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned",
    etc - They were added to document.
  4. But when I searched once again for "dataone" using java api, now
    the search is not returning the data.

how to make "data" field, should not be lost after updated - I
guess it lost but not sure, but not returning in the search result at
second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you could
as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #17

Perhaps others will chime in, but off the top of my head you have two
options:

  1. Fix the data in the application side before it is returned to the actual
    client. Are you returning straight JSON back to the client? Use Jackson
    (which is what elasticsearch also uses) to transform the JSON response.

  2. Use fields and not the source. Each field is indexed, but not stored by
    default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to use
the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

Ivan

On Fri, Nov 22, 2013 at 10:35 AM, Akram Hussain
akram.hussain.sk@gmail.comwrote:

Hi,

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo. :slight_smile:

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

Cheers,

Ivan

On Fri, Nov 22, 2013 at 10:19 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search first
time on the "data" field, but after updating, not able to search on the
"data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer
search against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from
other method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(searchCon
text.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());
QueryBuilder queryBuilder =
QueryBuilders.matchPhrasePrefixQuery("data", "dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuil
ders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain <akram.hu...@gmail.com

wrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

*
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field
is lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic
wrote:

It would help if you provided your mapping and an example query.
Are you searching against a certain field, the data field, or the _all
field? Create a gist as described here: http://www.elasticsearch
.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that some
updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to the
source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be
searchable.

  1. With the template I have created the doc like {"data" :
    "dataone", "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned",
    etc - They were added to document.
  4. But when I searched once again for "dataone" using java api,
    now the search is not returning the data.

how to make "data" field, should not be lost after updated - I
guess it lost but not sure, but not returning in the search result at
second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you
could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display _source.

Regards,
Akram.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(akram79) #18

Hi,

Can you provide me the reference api's how to achieve the mentioned ways.

Regards,
Akram.

On Saturday, November 23, 2013 12:27:46 AM UTC+5:30, Ivan Brusic wrote:

Perhaps others will chime in, but off the top of my head you have two
options:

  1. Fix the data in the application side before it is returned to the
    actual client. Are you returning straight JSON back to the client? Use
    Jackson (which is what elasticsearch also uses) to transform the JSON
    response.

  2. Use fields and not the source. Each field is indexed, but not stored by
    default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to
use the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

Ivan

On Fri, Nov 22, 2013 at 10:35 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo. :slight_smile:

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

Cheers,

Ivan

On Fri, Nov 22, 2013 at 10:19 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search
first time on the "data" field, but after updating, not able to search on
the "data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer
search against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from
other method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(searchCon
text.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.ge
tSize());
QueryBuilder queryBuilder =
QueryBuilders.matchPhrasePrefixQuery("data", "dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuil
ders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

*
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field
is lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic
wrote:

It would help if you provided your mapping and an example query.
Are you searching against a certain field, the data field, or the _all
field? Create a gist as described here: http://www.elasticsearch
.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that
some updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to
the source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be
searchable.

  1. With the template I have created the doc like {"data" :
    "dataone", "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned",
    etc - They were added to document.
  4. But when I searched once again for "dataone" using java api,
    now the search is not returning the data.

how to make "data" field, should not be lost after updated - I
guess it lost but not sure, but not returning in the search result at
second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you
could as
well include and exclude certain fields in the source:

"_source" : {
"includes" : ["timestamp", "title", "tinyfields."],
"excludes" : ["largefields.
"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display
_source.

Regards,
Akram.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Hannes Korte) #19

Hi Akram,

here are some links to achieve the approaches Ivan mentioned:

  1. If you don't expose the raw Elasticsearch API to your customers, then
    you can remove the sensitive information from the source JSON in your
    own application logic before you publish it. You may want to use a
    library like Jackson to work with JSON data:
    https://github.com/FasterXML/jackson

  2. Use separate fields and not the source. This way was described in
    Ivans other posts.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#string

But I fear it may not help you, if you want to be able to use the update
API without having to provide the sensitive information again. If this
information is not stored, due to exclusion from the source field or due
to a non-stored field, then you will have to provide it again when
updating it. All fields of a document have to be reindexed when
updating. If they are stored or part of the source they will be reused.
If they are not, they will be not be part of the new document, unless
they are provided with in the update.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html

So, I think the easiest way to achieve what you want is a possible third
solution: Simply exclude the sensitive data from the source and provide
it again along with every update. Good luck!

Best regards,
Hannes

On 23.11.2013 15:50, Akram Hussain wrote:

Hi,

Can you provide me the reference api's how to achieve the mentioned ways.

Regards,
Akram.

On Saturday, November 23, 2013 12:27:46 AM UTC+5:30, Ivan Brusic wrote:

Perhaps others will chime in, but off the top of my head you have two
options:

  1. Fix the data in the application side before it is returned to the
    actual client. Are you returning straight JSON back to the client? Use
    Jackson (which is what elasticsearch also uses) to transform the JSON
    response.

  2. Use fields and not the source. Each field is indexed, but not stored by
    default. Explicitly store the fields you want and retrieve them.

However, as you demonstrated, source needs to be enabled if you want to
use the update API. If you can skip the update API and simply reindex, the
issue is no longer relevant. If not, post-processing the result would be
your best bet. Maybe others would have some ideas.

Ivan

On Fri, Nov 22, 2013 at 10:35 AM, Akram Hussain <akram.hu...@gmail.com<javascript:>

wrote:

Hi,

As the "data" field is sensitive data, customer don't want it to be
visible as part of the source. But it should be searchable.

Let me know, if any way to achieve this.

Regards,
Akram.

On Friday, November 22, 2013 11:59:16 PM UTC+5:30, Ivan Brusic wrote:

Yes, I did ask before if that was a typo. :slight_smile:

An update in Lucene is basically an atomic delete and update.
Elasticsearch needs the source if it is to preserve the content between
updates. I would argue that the real bug here is that elasticsearch should
not have indexed the excluded field in the first place.

Perhaps the better question is what are you trying to solve by excluding
the field? Maybe you can leave the source intact and solve the original
issue in a different manner.

Cheers,

Ivan

On Fri, Nov 22, 2013 at 10:19 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

I was my once again copy & paste mistake.

{
"order_template" : {
"template" : "",
"order": 0,
"settings" :{
},
"mappings" : {
"default" : {
"_source" : {
"includes": ["
"],
"excludes": ["data"]
}
}
}
}
}

it is the data field, which was in "excludes" list, able to search
first time on the "data" field, but after updating, not able to search on
the "data" field.

I am sorry, once again for the confusion.

Regards,
Akram.

On Friday, November 22, 2013 11:30:33 PM UTC+5:30, Ivan Brusic wrote:

I do not see the issue on 0.90.5. After an update I can no longer
search against the "text" field, but "data" works fine.

On Fri, Nov 22, 2013 at 9:55 AM, Akram Hussain akram.hu...@gmail.comwrote:

Hi,

yes, I have missed the line, the code is, as it was returned from
other method, I missed it.

SearchRequestBuilder builder = client.prepareSearch(searchCon
text.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(searchContext.getOffset()).setSize(searchContext.ge
tSize());
QueryBuilder queryBuilder =
QueryBuilders.matchPhrasePrefixQuery("data", "dataValue");

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuil
ders.queryFilter(queryBuilder)));

The missed line is the bold line, it will search on "data" field
itself, after updating/adding new fields to the source, the field "data" is
getting lost from source.

Let me know the possible ways to handle it.

Regards,
Akram.

On Friday, November 22, 2013 11:14:36 PM UTC+5:30, Ivan Brusic wrote:

In your query code, you posted everything except the actual query!
Which field are you searching against? You mention that the data field is
lost, but you are excluding the text field. Typo?

I never exclude fields in source, but it seems like a field is
required if you want to search against it. It works after indexing, but
will disappear after updating. I doubt that is a bug, since elasticsearch
has nothing to update. An update basically is an atomic delete/insert.

Cheers,

Ivan

On Thu, Nov 21, 2013 at 8:21 PM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

The following one is my template

{

  • "order_template" : {*
  • "template" : "",
  • "order": 0,*
  • "settings" :{*
  • },*
  • "mappings" : {*
  • "default" : {*
  • "_source" : {*
  • "includes": [""],
  • "excludes": ["text"]*
  • }*
  • }*
  • }*
  • }*
    }

I am fetching the results using

SearchRequestBuilder builder =
client.prepareSearch(searchContext.getIndexName()).setSearchType(SearchType.DFS_QUERY_THEN_FETCH)

*
.setFrom(searchContext.getOffset()).setSize(searchContext.getSize());*

builder.setQuery(QueryBuilders.constantScoreQuery(FilterBuilders.queryFilter(queryBuilder)));

but updating is being done by following api

StringBuilder scriptBuilder = new StringBuilder();

scriptBuilder.append("ctx._source.").append("assigned").append("=").append("value");
client.prepareUpdate(index, type,
id).setScript(scriptBuilder.toString());

after updating with above api, the hidden field, i.e, *data *field
is lost.

I am doing any thing wrong while using api's.

Regards
Akram.

On Thursday, November 21, 2013 11:28:58 PM UTC+5:30, Ivan Brusic
wrote:

It would help if you provided your mapping and an example query.
Are you searching against a certain field, the data field, or the _all
field? Create a gist as described here: http://www.elasticsearch
.org/help/http://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fhelp%2F&sa=D&sntz=1&usg=AFQjCNGgo5_ert12o9OPxq7NdLYIOQ5Nig It
would help to retrieve the current mapping via the API, because some times
the mapping that is in effect might not be identical to the one you
supplied. :slight_smile:

One issue to point out, in case it was not understood, is that
some updates to the mapping will not affect documents that are currently
indexed. How are you adding fields to a document? Are you reindexing or
using the update API?

Cheers,

Ivan

On Thu, Nov 21, 2013 at 4:57 AM, Akram Hussain <
akram.hu...@gmail.com> wrote:

Hi,

Thanks for the reply.

It worked, but failed when try to add some other new fields to
the source from java api.

I have template like this

"mappings" : {
"default" : {
"_source" : {
"includes": ["*"],
"excludes": ["data"]
}
}
}

where only data field should be hidden, but it should be
searchable.

  1. With the template I have created the doc like {"data" :
    "dataone", "config": "config data", "status": "fine"}.
  2. When I searched for "dataone" using java api, it returned the
    correct data, with remaining fields displayed not displaying "data" field - it
    is fine
    .
  3. To the ssme document added some other fields like "assigned",
    etc - They were added to document.
  4. But when I searched once again for "dataone" using java api,
    now the search is not returning the data.

how to make "data" field, should not be lost after updated - I
guess it lost but not sure, but not returning in the search result at
second time).

Regards,
Akram.

On Thursday, November 21, 2013 1:58:39 PM UTC+5:30, Hannes Korte
wrote:

Hi Akram,

besides the solution of storing each field individually, you
could as
well include and exclude certain fields in the source:

"_source" : {
  "includes" : ["timestamp", "title", "tinyfields.*"],
  "excludes" : ["largefields.*"]
}

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/mapping-source-field.htmlhttp://www.google.com/url?q=http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-source-field.html&sa=D&sntz=1&usg=AFQjCNE15ilwN5QEN-j9w-sjFFzjpz0dUA

Best regards,
Hannes

On 20.11.2013 17:39, Akram Hussain wrote:

Hi,

I have some data, which should be hidden/not displayed as part
of the
_source.

when I set _source : { enabled : false}, source is being
returned as null.

How to make some fields are not displayed, but display
_source.

Regards,
Akram.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #20