# Indexing documents with complex \_id fields

**URL:** <https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056>\
**Category:** Elasticsearch\
**Created:** [August 3, 2013, 7:48pm UTC](https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056 "2013-08-03T19:48:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfrenkiel](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@sfrenkiel](https://discuss.elastic.co/u/sfrenkiel)\
**Post date:** [August 3, 2013, 7:48pm UTC](https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056/1 "2013-08-03T19:48:46Z")

</div>

I'm importing data from a mongodb store that uses subdocuments for the \_id  
field, and I'm having trouble searching against the data that I'm  
importing. I've broken down the problem into the following example:

# insert 2 documents, one with a complex \_id:

curl -XPOST "localhost:9200/index/test" -d '{"key":"value"}'  
curl -XPOST "localhost:9200/index/test" -d  
'{"\_id":{"name":"sfrenkiel"},"key":"value2"}}}'

# fetching the first works, of course:

curl "localhost:9200/index/test/\_search?pretty=true" -d  
'{"query":{"term":{"key":"value"}}}'  
{  
"took" : 2,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 5,  
"successful" : 5,  
"failed" : 0  
},  
"hits" : {  
"total" : 1,  
"max\_score" : 0.30685282,  
"hits" : [ {  
"\_index" : "index",  
"\_type" : "test",  
"\_id" : "A4AWuBXdQZqLxjxHK\_jk9g",  
"\_score" : 0.30685282, "\_source" : {"key":"value"}  
} ]  
}  
}

# but fetching the second does not:

curl "localhost:9200/index/test/\_search?pretty=true" -d  
'{"query":{"term":{"key":"value2"}}}'  
{  
"took" : 1,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 5,  
"successful" : 5,  
"failed" : 0  
},  
"hits" : {  
"total" : 0,  
"max\_score" : null,  
"hits" : []  
}  
}

# using the following to discover the list of indexed values reveals that

only the first value has been indexed (if I understand correctly what this  
command is doing):

curl -XGET "[http://localhost:9200/index/\_search?pretty=true](http://localhost:9200/index/_search?pretty=true)" -d '{"query":  
{"match\_al": {}}, "facets": {"tag": {"terms": {"field": "key"}}}, "size":  
0}'  
{  
"took" : 2,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 5,  
"successful" : 5,  
"failed" : 0  
},  
"hits" : {  
"total" : 2,  
"max\_score" : 1.0,  
"hits" : []  
},  
"facets" : {  
"tag" : {  
"\_type" : "terms",  
"missing" : 1,  
"total" : 1,  
"other" : 0,  
"terms" : [ {  
"term" : "value",  
"count" : 1  
} ]  
}  
}  
}

I've tried different mapping strategies and whatever options I can think  
of, and nothing seems to work. Strangely, entering the query to fetch the  
second record into the head plugin seems to work, but curl (and the java  
driver) do not. Can anyone help me out?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 5, 2013, 8:25am UTC](https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056/2 "2013-08-05T08:25:58Z")

</div>

Hey,

I havent tested anything, but I think if you

- specify \_id to select the ID in the document
- Do a POST without specifying an ID in the URL (which means the id is  
autogenerated)
- Have configured your ID mapping without a path, see  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/id-field/)

you might get this behaviour. Try recreating your index with the mapping  
configured correctly and I guess this works.

--Alex

On Sat, Aug 3, 2013 at 9:48 PM, Scott Frenkiel [sfrenkiel@gmail.com](mailto:sfrenkiel@gmail.com) wrote:

> I'm importing data from a mongodb store that uses subdocuments for the \_id  
> field, and I'm having trouble searching against the data that I'm  
> importing. I've broken down the problem into the following example:
> 
> # insert 2 documents, one with a complex \_id:
> 
> curl -XPOST "localhost:9200/index/test" -d '{"key":"value"}'  
> curl -XPOST "localhost:9200/index/test" -d  
> '{"\_id":{"name":"sfrenkiel"},"key":"value2"}}}'
> 
> # fetching the first works, of course:
> 
> curl "localhost:9200/index/test/\_search?pretty=true" -d  
> '{"query":{"term":{"key":"value"}}}'  
> {  
> "took" : 2,  
> "timed\_out" : false,  
> "\_shards" : {  
> "total" : 5,  
> "successful" : 5,  
> "failed" : 0  
> },  
> "hits" : {  
> "total" : 1,  
> "max\_score" : 0.30685282,  
> "hits" : [ {  
> "\_index" : "index",  
> "\_type" : "test",  
> "\_id" : "A4AWuBXdQZqLxjxHK\_jk9g",  
> "\_score" : 0.30685282, "\_source" : {"key":"value"}  
> } ]  
> }  
> }
> 
> # but fetching the second does not:
> 
> curl "localhost:9200/index/test/\_search?pretty=true" -d  
> '{"query":{"term":{"key":"value2"}}}'  
> {  
> "took" : 1,  
> "timed\_out" : false,  
> "\_shards" : {  
> "total" : 5,  
> "successful" : 5,  
> "failed" : 0  
> },  
> "hits" : {  
> "total" : 0,  
> "max\_score" : null,  
> "hits" :   
> }  
> }
> 
> # using the following to discover the list of indexed values reveals that
> 
> only the first value has been indexed (if I understand correctly what this  
> command is doing):
> 
> curl -XGET "[http://localhost:9200/index/\_search?pretty=true](http://localhost:9200/index/_search?pretty=true)" -d  
> '{"query": {"match\_al": {}}, "facets": {"tag": {"terms": {"field":  
> "key"}}}, "size": 0}'  
> {  
> "took" : 2,  
> "timed\_out" : false,  
> "\_shards" : {  
> "total" : 5,  
> "successful" : 5,  
> "failed" : 0  
> },  
> "hits" : {  
> "total" : 2,  
> "max\_score" : 1.0,  
> "hits" :   
> },  
> "facets" : {  
> "tag" : {  
> "\_type" : "terms",  
> "missing" : 1,  
> "total" : 1,  
> "other" : 0,  
> "terms" : [ {  
> "term" : "value",  
> "count" : 1  
> } ]  
> }  
> }  
> }
> 
> I've tried different mapping strategies and whatever options I can think  
> of, and nothing seems to work. Strangely, entering the query to fetch the  
> second record into the head plugin seems to work, but curl (and the java  
> driver) do not. Can anyone help me out?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![sfrenkiel](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@sfrenkiel](https://discuss.elastic.co/u/sfrenkiel)\
**Post date:** [August 5, 2013, 2:50pm UTC](https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056/3 "2013-08-05T14:50:34Z")

</div>

Thanks for responding.

> - Do a POST without specifying an ID in the URL (which means the id is  
> autogenerated)  
> I'm uncertain how to insert a record specifying the \_id field in the URL  
> when the \_id field is a subdocument anyway. So POST is the only method I  
> know.

> - Have configured your ID mapping without a path, see  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/id-field/)  
> I tried the following mapping:

{ "mappings": { "type" : { "properties" : { "\_id" : { "path" : "\_id" } } }  
} }

And got an error "Trying to parse an object but has a different type". No  
idea what that means.

--

Your post prompted me to look more closely at how my two documents are  
stored. The one with the complex id looks like:

```
"hits" : [ {
  "_index" : "index",
  "_type" : "test",
  "_id" : "oPC7SzPXTEa8wT9iZ6S5gw",
  "_score" : 1.0, "_source" :

```

{"\_id":{"name":"sfrenkiel"},"key":"value2"}}}  
}, {

So I notice that the \_id field in the document source is different from the  
top-level \_id. When using the mongodb river, I see that the top-level \_id  
field and the \_id from the source are the same. Regardless, I see the same  
problematic behavior in both cases (other fields in the original source doc  
are not indexed).

For those not familiar with mongodb, they use the same \_id key to represent  
primary keys. So is there something special about inserting a record that  
has a complex \_id field. As an additional test, I inserted a doc with a  
complex field NOT called \_id, and was able to query the other fields in the  
document just fine.

Thanks,  
Scott

On Mon, Aug 5, 2013 at 4:25 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote:

> Hey,
> 
> I havent tested anything, but I think if you
> 
> - specify \_id to select the ID in the document
> - Do a POST without specifying an ID in the URL (which means the id is  
> autogenerated)
> - Have configured your ID mapping without a path, see  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/id-field/)
> 
> you might get this behaviour. Try recreating your index with the mapping  
> configured correctly and I guess this works.
> 
> --Alex
> 
> On Sat, Aug 3, 2013 at 9:48 PM, Scott Frenkiel [sfrenkiel@gmail.com](mailto:sfrenkiel@gmail.com)wrote:
> 
> > I'm importing data from a mongodb store that uses subdocuments for the  
> > \_id field, and I'm having trouble searching against the data that I'm  
> > importing. I've broken down the problem into the following example:
> > 
> > # insert 2 documents, one with a complex \_id:
> > 
> > curl -XPOST "localhost:9200/index/test" -d '{"key":"value"}'  
> > curl -XPOST "localhost:9200/index/test" -d  
> > '{"\_id":{"name":"sfrenkiel"},"key":"value2"}}}'
> > 
> > # fetching the first works, of course:
> > 
> > curl "localhost:9200/index/test/\_search?pretty=true" -d  
> > '{"query":{"term":{"key":"value"}}}'  
> > {  
> > "took" : 2,  
> > "timed\_out" : false,  
> > "\_shards" : {  
> > "total" : 5,  
> > "successful" : 5,  
> > "failed" : 0  
> > },  
> > "hits" : {  
> > "total" : 1,  
> > "max\_score" : 0.30685282,  
> > "hits" : [ {  
> > "\_index" : "index",  
> > "\_type" : "test",  
> > "\_id" : "A4AWuBXdQZqLxjxHK\_jk9g",  
> > "\_score" : 0.30685282, "\_source" : {"key":"value"}  
> > } ]  
> > }  
> > }
> > 
> > # but fetching the second does not:
> > 
> > curl "localhost:9200/index/test/\_search?pretty=true" -d  
> > '{"query":{"term":{"key":"value2"}}}'  
> > {  
> > "took" : 1,  
> > "timed\_out" : false,  
> > "\_shards" : {  
> > "total" : 5,  
> > "successful" : 5,  
> > "failed" : 0  
> > },  
> > "hits" : {  
> > "total" : 0,  
> > "max\_score" : null,  
> > "hits" :   
> > }  
> > }
> > 
> > # using the following to discover the list of indexed values reveals that
> > 
> > only the first value has been indexed (if I understand correctly what this  
> > command is doing):
> > 
> > curl -XGET "[http://localhost:9200/index/\_search?pretty=true](http://localhost:9200/index/_search?pretty=true)" -d  
> > '{"query": {"match\_al": {}}, "facets": {"tag": {"terms": {"field":  
> > "key"}}}, "size": 0}'  
> > {  
> > "took" : 2,  
> > "timed\_out" : false,  
> > "\_shards" : {  
> > "total" : 5,  
> > "successful" : 5,  
> > "failed" : 0  
> > },  
> > "hits" : {  
> > "total" : 2,  
> > "max\_score" : 1.0,  
> > "hits" :   
> > },  
> > "facets" : {  
> > "tag" : {  
> > "\_type" : "terms",  
> > "missing" : 1,  
> > "total" : 1,  
> > "other" : 0,  
> > "terms" : [ {  
> > "term" : "value",  
> > "count" : 1  
> > } ]  
> > }  
> > }  
> > }
> > 
> > I've tried different mapping strategies and whatever options I can think  
> > of, and nothing seems to work. Strangely, entering the query to fetch the  
> > second record into the head plugin seems to work, but curl (and the java  
> > driver) do not. Can anyone help me out?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > 
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit  
> [https://groups.google.com/d/topic/elasticsearch/0at1uZBvN3k/unsubscribe](https://groups.google.com/d/topic/elasticsearch/0at1uZBvN3k/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
Scott Frenkiel  
(732) 239-5012  
[sfrenkiel@gmail.com](mailto:sfrenkiel@gmail.com)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:22am UTC](https://discuss.elastic.co/t/indexing-documents-with-complex--id-fields/13056/4 "2017-07-06T02:22:52Z")

</div>