# Inconsistent 'fields' returned when explicitly asking for fields in 0.19.4 **URL:** \ **Category:** Elasticsearch\ **Created:** [February 27, 2013, 8:06pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934 "2013-02-27T20:06:44Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [February 27, 2013, 8:06pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/1 "2013-02-27T20:06:44Z")
Hi, all. We have an odd case where requesting explicit fields on a query sometimes returns all the fields, and sometimes doesn't - for the same ids query (i.e. for the same record). We haven't been able to nail it down - but it _usually_ works fine if we've just re-indexed and haven't yet shut down and restarted ES, and _usually_ doesn't work if we've shut down and restarted ES. We don't have any stored fields - everything is in the \_source. So - here's a CURL sample that sometimes returns the requested 'blob' field and sometimes doesn't. (I may mess up the formatting here - I'm using CURL on Windows, and taking my original Windows command and morphing it into something more readable: curl -XGET [http://localhost:9200/cust1/\_search?fields=blob](http://localhost:9200/cust1/_search?fields=blob) -d '{ "query" : { "ids" : { "values" : ["ent://SD\_ASSET/0/122"] } } }' When it DOES NOT return the blob field, here's the resulting JSON: { "\_shards" : { "failed" : 0, "successful" : 1, "total" : 1 }, "hits" : { "hits" : [ { "\_id" : "ent://SD\_ASSET/0/122", "\_index" : "cust1", "\_score" : 1.0, "\_type" : "sd\_asset" } ], "max\_score" : 1.0, "total" : 1 }, "timed\_out" : false, "took" : 1 } When it DOES return the blob field, here's the resulting JSON: { "\_shards" : { "failed" : 0, "successful" : 1, "total" : 1 }, "hits" : { "hits" : [ { "\_id" : "ent://SD\_ASSET/0/122", "\_index" : "cust1", "\_score" : 1.0, "\_type" : "sd\_asset", "fields" : { "\_source.blob" : "H4sIAAAAAAAAAJ3UUWvDIBAA4L/Sx+1lLun2UJAQMVcqTTSorGQvEhrXBUwGW7Pfv3QZY82KBZ+8Qz8PzkPcvO2HzvbHBLdNEsUxRuOKu/q4f53SKcQvrXXjRl93NtlCtRMyUxh9p/izdoNN+sE5jKYYo5/zf5kEpSWjmgk+o2kKRamrNPV6QikoZXJ4gvzcR4EuDnTLQPfgdWshC6LPBVEK9JWujoUIp2CUkDO98krKNHsGrjasDHoPmgORp8qzpt6HqCAUBanVlbZcVo9exAEyo4XZcrELamYBGSNGV+WsrBF6A9L4B+40JIaTYmaVfW9r97E4DG1jFzfLu+j23z3o9wP4AlqyXlULBAAA" } } ], "max\_score" : 1.0, "total" : 1 }, "timed\_out" : false, "took" : 2 } If I don't ask for any fields, here's the response (just to show you what's in the \_source) { "\_shards" : { "failed" : 0, "successful" : 1, "total" : 1 }, "hits" : { "hits" : [ { "\_id" : "ent://SD\_ASSET/0/122", "\_index" : "cust1", "\_score" : 1.0, "\_source" : { "ACCESS\_LEVEL\_nfacet" : [ "1", "2", "3", "4" ], "ASSET\_NAME\_display" : ["Serials guide (3.1)"], "CITIZENSHIPS\_facet" : ["@@EMPTY@@"], "CLEARANCE\_nfacet" : [ "20", "0", "10", "99", "5" ], "DOC\_TEXT" : [ "null", "ASSET", "_OTHER_", "Serials guide (3.1)" ], "DS\_EC" : "SD\_ASSET", "DS\_KEY" : 7, "FORMAT\_display" : ["ASSET"], "FORMAT\_facet" : ["ASSET"], "KEYWORDS\_display" : ["null"], "KEYWORDS\_facet" : ["null"], "MEDIA\_TYPE\_display" : ["_OTHER_"], "MEDIA\_TYPE\_facet" : ["_OTHER_"], "NEED\_TO\_KNOWS\_facet" : ["@@EMPTY@@"], "RELEVANCE\_SORT\_nsort" : "9", "RESTRICTIONS\_facet" : ["@@EMPTY@@"], "\_id" : "ent://SD\_ASSET/0/122", "blob" : "H4sIAAAAAAAAAJ3UUWvDIBAA4L/Sx+1lLun2UJAQMVcqTTSorGQvEhrXBUwGW7Pfv3QZY82KBZ+8Qz8PzkPcvO2HzvbHBLdNEsUxRuOKu/q4f53SKcQvrXXjRl93NtlCtRMyUxh9p/izdoNN+sE5jKYYo5/zf5kEpSWjmgk+o2kKRamrNPV6QikoZXJ4gvzcR4EuDnTLQPfgdWshC6LPBVEK9JWujoUIp2CUkDO98krKNHsGrjasDHoPmgORp8qzpt6HqCAUBanVlbZcVo9exAEyo4XZcrELamYBGSNGV+WsrBF6A9L4B+40JIaTYmaVfW9r97E4DG1jFzfLu+j23z3o9wP4AlqyXlULBAAA", "sequence" : 4 }, "\_type" : "sd\_asset" } ], "max\_score" : 1.0, "total" : 1 }, "timed\_out" : false, "took" : 1 } Anyways - this is a sort of minimal example. What we're really seeing is on a request for 6 fields to be returned, sometimes we get all 6, sometimes we just get one or two. I know that we can modify the query to prefix all the field names by "\_source.", and that does indeed consistently return all the requested fields. But, since my understanding is that when requesting fields just by name, they should be returned from the \_source without needing that qualifier. Any ideas if this is a bug? I looked through the archives and found one that's similar, but not identical: [https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/fields$20not$20returned/elasticsearch/z\_Er\_Q1N1yw/tWb1JfvuaHUJ](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/fields%2420not%2420returned/elasticsearch/z_Er_Q1N1yw/tWb1JfvuaHUJ) (That's where I found the idea of qualifying the field names) I've attached the index meta data if that is of any help. Thanks! Bob. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [February 27, 2013, 10:43pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/2 "2013-02-27T22:43:51Z")
I haven't used this much, but here's what I've read: 1. ElasticSearch stores and retrieves your \_source JSON without modification. I've read that it doesn't even need to be fully valid, though I have not tried that! 2. To be returned explicitly by a request of a list of fields, the field must be stored. Contained in the \_source doesn't count, I don't believe. At least, not from my experience. 3. Also, to be sorted on, the field must be stored, and it's recommended that a second copy must also be stored without being broken into tokens. Though for now, I only sort on Geo distance, and after following all of the examples it does work very nicely. Since I typically parse the returned \_source and it already contains the entire document, I don't bother trying to tell ElasticSearch what fields I want. I pull what I want from my parsed JSON. So I haven't explored the added benefits of actually storing fields, nor of storing two copies: One analyzed and one not analyzed. That may be next on my list of cool ES features to dive into. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [February 28, 2013, 1:44pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/3 "2013-02-28T13:44:03Z")
Thanks, but that's not the issue. The issue is for the same query, sometimes the results come back fine, and sometimes they don't. Identical source data, identical indexing, identical index - just shutting down ES and starting it up again can change the behaviour... ☹ 1. Right, it does. 2. On this page: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) the documentation indicates that if a field isn't stored, it will load the \_source and extract the field from there. 3. Adding stored = true to the fields will both store it in the index, AND have it in the \_source. One or the other is fine for the fields I care about - I don't want both - just bloats the index size. On Wednesday, 27 February 2013 17:43:51 UTC-5, InquiringMind wrote: > I haven't used this much, but here's what I've read: > > 1. Elasticsearch stores and retrieves your \_source JSON without > modification. I've read that it doesn't even need to be fully valid, though > I have not tried that! > > 2. To be returned explicitly by a request of a list of fields, the field > must be stored. Contained in the \_source doesn't count, I don't believe. At > least, not from my experience. > > 3. Also, to be sorted on, the field must be stored, and it's recommended > that a second copy must also be stored without being broken into tokens. > Though for now, I only sort on Geo distance, and after following all of the > examples it does work very nicely. > > Since I typically parse the returned \_source and it already contains the > entire document, I don't bother trying to tell Elasticsearch what fields I > want. I pull what I want from my parsed JSON. > > So I haven't explored the added benefits of actually storing fields, nor > of storing two copies: One analyzed and one not analyzed. That may be next > on my list of cool ES features to dive into. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 28, 2013, 9:23pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/4 "2013-02-28T21:23:11Z")
You do not need to store the field, reading from source works as well. The default Elasticsearch behavior of not-storing individual fields, but store the original source works well in most cases. If you are requesting six fields, you are better off using source instead of having to do six different field lookups. Although the OP didn't mention sorting, a field must be indexed, not stored although it can be, in order to be sorted on. Shot in the dark here: you mentioned that it works after a reindex. Are you perhaps indexing an incorrect document in between re-indexes? The "no fields" example above, was that taken after a failed query? - Ivan On Thu, Feb 28, 2013 at 5:44 AM, Robert Sandiford [bobsandiford@gmail.com](mailto:bobsandiford@gmail.com)wrote: > Thanks, but that's not the issue. The issue is for the same query, > sometimes the results come back fine, and sometimes they don't. Identical > source data, identical indexing, identical index - just shutting down ES > and starting it up again can change the behaviour... ☹ > > 1. Right, it does. > 2. On this page: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) the > documentation indicates that if a field isn't stored, it will load the > \_source and extract the field from there. > 3. Adding stored = true to the fields will both store it in the index, AND > have it in the \_source. One or the other is fine for the fields I care > about - I don't want both - just bloats the index size. > > On Wednesday, 27 February 2013 17:43:51 UTC-5, InquiringMind wrote: > > > I haven't used this much, but here's what I've read: > > > > 1. Elasticsearch stores and retrieves your \_source JSON without > > modification. I've read that it doesn't even need to be fully valid, though > > I have not tried that! > > > > 2. To be returned explicitly by a request of a list of fields, the field > > must be stored. Contained in the \_source doesn't count, I don't believe. At > > least, not from my experience. > > > > 3. Also, to be sorted on, the field must be stored, and it's recommended > > that a second copy must also be stored without being broken into tokens. > > Though for now, I only sort on Geo distance, and after following all of the > > examples it does work very nicely. > > > > Since I typically parse the returned \_source and it already contains the > > entire document, I don't bother trying to tell Elasticsearch what fields I > > want. I pull what I want from my parsed JSON. > > > > So I haven't explored the added benefits of actually storing fields, nor > > of storing two copies: One analyzed and one not analyzed. That may be next > > on my list of cool ES features to dive into. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [February 28, 2013, 10:04pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/5 "2013-02-28T22:04:44Z")
Thanks, Ivan, for confirming my understanding that I don't need to flag the fields as stored - just having them in the \_source is sufficient. It's not the indexing itself. We're using the exact same input data and code to index each time. What I'm seeing is that (generally) once I've indexed, it works fine, until I shut down and restart ES. After the restart, (generally) it won't work. I.E. no reindex in between, just stopping and starting ES. Bob. On Thursday, 28 February 2013 16:23:11 UTC-5, Ivan Brusic wrote: > You do not need to store the field, reading from source works as well. The > default Elasticsearch behavior of not-storing individual fields, but store > the original source works well in most cases. If you are requesting six > fields, you are better off using source instead of having to do six > different field lookups. Although the OP didn't mention sorting, a field > must be indexed, not stored although it can be, in order to be sorted on. > > Shot in the dark here: you mentioned that it works after a reindex. Are > you perhaps indexing an incorrect document in between re-indexes? The "no > fields" example above, was that taken after a failed query? > > - > > Ivan > > On Thu, Feb 28, 2013 at 5:44 AM, Robert Sandiford \<[bobsan...@gmail.com](mailto:bobsan...@gmail.com)\ > > > wrote: > > > Thanks, but that's not the issue. The issue is for the same query, > > sometimes the results come back fine, and sometimes they don't. Identical > > source data, identical indexing, identical index - just shutting down ES > > and starting it up again can change the behaviour... ☹ > > > > 1. Right, it does. > > 2. On this page: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) the > > documentation indicates that if a field isn't stored, it will load the > > \_source and extract the field from there. > > 3. Adding stored = true to the fields will both store it in the index, > > AND have it in the \_source. One or the other is fine for the fields I care > > about - I don't want both - just bloats the index size. > > > > On Wednesday, 27 February 2013 17:43:51 UTC-5, InquiringMind wrote: > > > > > I haven't used this much, but here's what I've read: > > > > > > 1. Elasticsearch stores and retrieves your \_source JSON without > > > modification. I've read that it doesn't even need to be fully valid, though > > > I have not tried that! > > > > > > 2. To be returned explicitly by a request of a list of fields, the field > > > must be stored. Contained in the \_source doesn't count, I don't believe. At > > > least, not from my experience. > > > > > > 3. Also, to be sorted on, the field must be stored, and it's recommended > > > that a second copy must also be stored without being broken into tokens. > > > Though for now, I only sort on Geo distance, and after following all of the > > > examples it does work very nicely. > > > > > > Since I typically parse the returned \_source and it already contains the > > > entire document, I don't bother trying to tell Elasticsearch what fields I > > > want. I pull what I want from my parsed JSON. > > > > > > So I haven't explored the added benefits of actually storing fields, nor > > > of storing two copies: One analyzed and one not analyzed. That may be next > > > on my list of cool ES features to dive into. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 1, 2013, 11:52am UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/6 "2013-03-01T11:52:24Z")
Hi Bob On Thu, 2013-02-28 at 14:04 -0800, Robert Sandiford wrote: > Thanks, Ivan, for confirming my understanding that I don't need to > flag the fields as stored - just having them in the \_source is > sufficient. > > It's not the indexing itself. We're using the exact same input data > and code to index each time. > > What I'm seeing is that (generally) once I've indexed, it works fine, > until I shut down and restart ES. After the restart, (generally) it > won't work. I.E. no reindex in between, just stopping and starting > ES. Is it possible that you have multiple fields with the same name? A structure like ``` { foo: { bar: { baz: "xxx"}, baz: "zzz" }} ``` might choose "foo.bar.baz" or "foo.baz" on different runs, and so confuse the results. Note: the same field name in different types can have a similar effect. clint > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [March 1, 2013, 1:02pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/7 "2013-03-01T13:02:02Z")
Nope, no duplicate fields (and, no structures - just simple, single-level fields). There's a complete copy of the contents of an index result (i.e. no fields specified - shows the \_source contents) in the original post. The only potential 'duplicate' is that the \_id field exists both inside and outside of the \_source. On Friday, 1 March 2013 06:52:24 UTC-5, Clinton Gormley wrote: > Hi Bob > > On Thu, 2013-02-28 at 14:04 -0800, Robert Sandiford wrote: > > > Thanks, Ivan, for confirming my understanding that I don't need to > > flag the fields as stored - just having them in the \_source is > > sufficient. > > > > It's not the indexing itself. We're using the exact same input data > > and code to index each time. > > > > What I'm seeing is that (generally) once I've indexed, it works fine, > > until I shut down and restart ES. After the restart, (generally) it > > won't work. I.E. no reindex in between, just stopping and starting > > ES. > > Is it possible that you have multiple fields with the same name? > > A structure like > > ``` > { foo: { > bar: { baz: "xxx"}, > baz: "zzz" > }} > > ``` > > might choose "foo.bar.baz" or "foo.baz" on different runs, and so > confuse the results. Note: the same field name in different types can > have a similar effect. > > clint > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 1, 2013, 1:12pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/8 "2013-03-01T13:12:01Z")
On Fri, 2013-03-01 at 05:02 -0800, Robert Sandiford wrote: > Nope, no duplicate fields (and, no structures - just simple, > single-level fields). There's a complete copy of the contents of an > index result (i.e. no fields specified - shows the \_source contents) > in the original post. The only potential 'duplicate' is that the \_id > field exists both inside and outside of the \_source. Any chance you could put together a recreation? Also, are you using the latest version of ES? 0.20.5? clint > On Friday, 1 March 2013 06:52:24 UTC-5, Clinton Gormley wrote: > Hi Bob > > ``` > On Thu, 2013-02-28 at 14:04 -0800, Robert Sandiford wrote: > > Thanks, Ivan, for confirming my understanding that I don't > need to > > flag the fields as stored - just having them in the _source > is > > sufficient. > > > > It's not the indexing itself. We're using the exact same > input data > > and code to index each time. > > > > What I'm seeing is that (generally) once I've indexed, it > works fine, > > until I shut down and restart ES. After the restart, > (generally) it > > won't work. I.E. no reindex in between, just stopping and > starting > > ES. > > Is it possible that you have multiple fields with the same > name? > > A structure like > > { foo: { > bar: { baz: "xxx"}, > baz: "zzz" > }} > > might choose "foo.bar.baz" or "foo.baz" on different runs, and > so > confuse the results. Note: the same field name in different > types can > have a similar effect. > > clint > > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [March 1, 2013, 1:51pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/9 "2013-03-01T13:51:56Z")
HI, Clint. More info is in the original post - but, we're running 0.19.4. We were hoping to wait for 0.90 to be GA before updating to a more recent version - i.e. we want to upgrade once, from 0.19.4 to 0.90. (We have some custom Lucene code we roll in - which is NOT in the area of retrieving fields (we mess with similarity a bit, and do some custom facet filtering in some situations - not this situation) As to a recreation, I'll try to figure one out - but I haven't found a consistent way to reproduce it - it's intermittent. On Friday, 1 March 2013 08:12:01 UTC-5, Clinton Gormley wrote: > On Fri, 2013-03-01 at 05:02 -0800, Robert Sandiford wrote: > > > Nope, no duplicate fields (and, no structures - just simple, > > single-level fields). There's a complete copy of the contents of an > > index result (i.e. no fields specified - shows the \_source contents) > > in the original post. The only potential 'duplicate' is that the \_id > > field exists both inside and outside of the \_source. > > Any chance you could put together a recreation? > > Also, are you using the latest version of ES? 0.20.5? > > clint > > > On Friday, 1 March 2013 06:52:24 UTC-5, Clinton Gormley wrote: > > Hi Bob > > > > ``` > > On Thu, 2013-02-28 at 14:04 -0800, Robert Sandiford wrote: > > > Thanks, Ivan, for confirming my understanding that I don't > > need to > > > flag the fields as stored - just having them in the _source > > is > > > sufficient. > > > > > > It's not the indexing itself. We're using the exact same > > input data > > > and code to index each time. > > > > > > What I'm seeing is that (generally) once I've indexed, it > > works fine, > > > until I shut down and restart ES. After the restart, > > (generally) it > > > won't work. I.E. no reindex in between, just stopping and > > starting > > > ES. > > > > Is it possible that you have multiple fields with the same > > name? > > > > A structure like > > > > { foo: { > > bar: { baz: "xxx"}, > > baz: "zzz" > > }} > > > > might choose "foo.bar.baz" or "foo.baz" on different runs, and > > so > > confuse the results. Note: the same field name in different > > types can > > have a similar effect. > > > > clint > > > > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [March 1, 2013, 5:37pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/10 "2013-03-01T17:37:51Z")
Be careful not to mix Lucene libraries in your code. Your existing code would also need to be updated to Lucene 4.1. I am the the process in converting now and I finally can no longer ignore all the deprecation warnings! 🙂 -- Ivan On Fri, Mar 1, 2013 at 5:51 AM, Robert Sandiford [bobsandiford@gmail.com](mailto:bobsandiford@gmail.com)wrote: > HI, Clint. > > More info is in the original post - but, we're running 0.19.4. We were > hoping to wait for 0.90 to be GA before updating to a more recent version - > i.e. we want to upgrade once, from 0.19.4 to 0.90. (We have some custom > Lucene code we roll in - which is NOT in the area of retrieving fields (we > mess with similarity a bit, and do some custom facet filtering in some > situations - not this situation) > > As to a recreation, I'll try to figure one out - but I haven't found a > consistent way to reproduce it - it's intermittent. > > On Friday, 1 March 2013 08:12:01 UTC-5, Clinton Gormley wrote: > > > On Fri, 2013-03-01 at 05:02 -0800, Robert Sandiford wrote: > > > > > Nope, no duplicate fields (and, no structures - just simple, > > > single-level fields). There's a complete copy of the contents of an > > > index result (i.e. no fields specified - shows the \_source contents) > > > in the original post. The only potential 'duplicate' is that the \_id > > > field exists both inside and outside of the \_source. > > > > Any chance you could put together a recreation? > > > > Also, are you using the latest version of ES? 0.20.5? > > > > clint > > > > > On Friday, 1 March 2013 06:52:24 UTC-5, Clinton Gormley wrote: > > > Hi Bob > > > > > > ``` > > > On Thu, 2013-02-28 at 14:04 -0800, Robert Sandiford wrote: > > > > Thanks, Ivan, for confirming my understanding that I don't > > > need to > > > > flag the fields as stored - just having them in the _source > > > is > > > > sufficient. > > > > > > > > It's not the indexing itself. We're using the exact same > > > input data > > > > and code to index each time. > > > > > > > > What I'm seeing is that (generally) once I've indexed, it > > > works fine, > > > > until I shut down and restart ES. After the restart, > > > (generally) it > > > > won't work. I.E. no reindex in between, just stopping and > > > starting > > > > ES. > > > > > > Is it possible that you have multiple fields with the same > > > name? > > > > > > A structure like > > > > > > { foo: { > > > bar: { baz: "xxx"}, > > > baz: "zzz" > > > }} > > > > > > might choose "foo.bar.baz" or "foo.baz" on different runs, and > > > so > > > confuse the results. Note: the same field name in different > > > types can > > > have a similar effect. > > > > > > clint > > > > > > > > > > > > > ``` > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [March 7, 2013, 4:10pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/11 "2013-03-07T16:10:18Z")
I figured out the bug in 0.19.4. Basically, we have mappings like this (in part): ``` "mappings" : { "_default_" : { "properties" : { "blob" : {"type" : "binary", "store" : "false", "index" : ``` "not\_analyzed", "include\_in\_all" : "false" } } } } When starting 'from scratch' (i.e. indexing data for the first time), internally ES has BinaryFieldMapper element for "blob", where "store" is "NO", which is correct. However - when shutting down and re-starting ES, the BinaryFieldMapper entry that gets created has "store" set to "YES". So, it tries in FetchPhase to place the "blob" info into the fieldSelectorMapper, rather than into the extractFieldNames. And, since there is no "blob" field other than in \_source, it fails to retrieve the "blob" value. So - the but appears to be that the "store" "false" is not being preserved properly when you just shut down ES and then start it up again. I'm going to leave this to those who know the ES internals to take a look at this particular bug. Thanks! Bob. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 7, 2013, 6:31pm UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/12 "2013-03-07T18:31:39Z")
On Thu, 2013-03-07 at 08:10 -0800, Robert Sandiford wrote: > I figured out the bug in 0.19.4. Nice catch. Please open an issue for this, so that it doesn't get lost ta clint > Basically, we have mappings like this (in part): > > ``` > "mappings" : { > "_default_" : { > "properties" : { > "blob" : {"type" : "binary", "store" : "false", > > ``` > > "index" : "not\_analyzed", "include\_in\_all" : "false" } > } > } > } > > When starting 'from scratch' (i.e. indexing data for the first time), > internally ES has BinaryFieldMapper element for "blob", where "store" > is "NO", which is correct. > > However - when shutting down and re-starting ES, the BinaryFieldMapper > entry that gets created has "store" set to "YES". So, it tries in > FetchPhase to place the "blob" info into the fieldSelectorMapper, > rather than into the extractFieldNames. And, since there is no "blob" > field other than in \_source, it fails to retrieve the "blob" value. > > So - the but appears to be that the "store" "false" is not being > preserved properly when you just shut down ES and then start it up > again. > > I'm going to leave this to those who know the ES internals to take a > look at this particular bug. > > Thanks! > > Bob. > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [March 8, 2013, 1:54am UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/13 "2013-03-08T01:54:56Z")
Turns out it has been fixed sometime post-0.19.4. Works fine (differently, but fine :)) in both 0.20.5 and 0.90.0 Beta 1. (Once I figured out how to run an "ids" query - another bug, resolved but not in Beta 1). Thanks, all! Bob. On Thursday, 7 March 2013 13:31:39 UTC-5, Clinton Gormley wrote: > On Thu, 2013-03-07 at 08:10 -0800, Robert Sandiford wrote: > > > I figured out the bug in 0.19.4. > > Nice catch. > > Please open an issue for this, so that it doesn't get lost > > ta > > clint > > > Basically, we have mappings like this (in part): > > > > ``` > > "mappings" : { > > "_default_" : { > > "properties" : { > > "blob" : {"type" : "binary", "store" : "false", > > > > ``` > > > > "index" : "not\_analyzed", "include\_in\_all" : "false" } > > } > > } > > } > > > > When starting 'from scratch' (i.e. indexing data for the first time), > > internally ES has BinaryFieldMapper element for "blob", where "store" > > is "NO", which is correct. > > > > However - when shutting down and re-starting ES, the BinaryFieldMapper > > entry that gets created has "store" set to "YES". So, it tries in > > FetchPhase to place the "blob" info into the fieldSelectorMapper, > > rather than into the extractFieldNames. And, since there is no "blob" > > field other than in \_source, it fails to retrieve the "blob" value. > > > > So - the but appears to be that the "store" "false" is not being > > preserved properly when you just shut down ES and then start it up > > again. > > > > I'm going to leave this to those who know the ES internals to take a > > look at this particular bug. > > > > Thanks! > > > > Bob. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/inconsistent-fields-returned-when-explicitly-asking-for-fields-in-0-19-4/10934/14 "2017-07-06T02:47:41Z")