# Parent-Child Indices Getting Out of Sync **URL:** \ **Category:** Elasticsearch\ **Created:** [October 12, 2018, 5:33pm UTC](https://discuss.elastic.co/t/parent-child-indices-getting-out-of-sync/152281 "2018-10-12T17:33:27Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![TheConnMan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theconnman/32/36487_2.png) [@TheConnMan](https://discuss.elastic.co/u/TheConnMan)\ **Post date:** [October 12, 2018, 5:33pm UTC](https://discuss.elastic.co/t/parent-child-indices-getting-out-of-sync/152281/1 "2018-10-12T17:33:27Z")
Environment: 4 nodes, 124GB RAM total, ~.5TB of data, v5.6.8 Indices: 1 main index, 1 parent doc type, 3 child doc types, ~300 index operations/s on the primary shards Clients: Primarily the Java SDK ## Issue Symptoms Our core index has a parent doc type (elements) and a secondary core type (metrics) which is a child document (linked by a primary key). We have other child doc types, but one is enough to illustrate the issue. Metrics are also embedded docs to elements, but they are duplicated to their own doc type so we can query and fetch metrics matching criteria, not just elements _with_ metrics matching criteria. We've found that metrics in the child doc index go missing. I've done _a lot_ of research into this to try to replicate. The two scenarios I've tried to replicate are: - Are we getting bulk index failures when saving metrics? I log them and can't find a single one. I _can_ replicate errors in a test environment, but that's through maxing out the bulk index thread pool. - Are metrics getting indexed on a primary shard, not yet replicated to secondary shards, then the primary fails over? I've tried replicating this as well with a small Dockerized cluster without success. ## Questions I've got two main lines of questions: - Does anyone have any advice on other things to try? Have you seen a scenario like this before? Is the Java SDK lying to me about bulk index errors? Is nothing possibly wrong with Elasticsearch and I need to dig into my app code more? - I understand duplicating data like this is not great. I don't like it. Do you know of a better way to model the data so I can pull back child documents from queries, but only storing them as embedded documents? Thanks in advance, I'm happy to provide more detail if needed. ---
**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\ **Post date:** [October 14, 2018, 3:03am UTC](https://discuss.elastic.co/t/parent-child-indices-getting-out-of-sync/152281/2 "2018-10-14T03:03:07Z")
Hi Brian, > [@TheConnMan](#): > > - Does anyone have any advice on other things to try? Have you seen a scenario like this before? Is the Java SDK lying to me about bulk index errors? Is nothing possibly wrong with Elasticsearch and I need to dig into my app code more? When you asking about it lying for bulk indexing errors, what errors are you talking about? Something to check is whether you are indexing any of the parent documents with custom routing values, in the event that you index the parent type with a custom routing and then use the `_id` (instead of the routing value) for the child type, they will end up in different shards and the `has_child` query won't find the document. > - I understand duplicating data like this is not great. I don't like it. Do you know of a better way to model the data so I can pull back child documents from queries, but only storing them as embedded documents? You may be able to return the documents that you want without duplicating them into the parent document by using the `inner_hits` parameter, see: [Inner hits | Elasticsearch Guide [6.4] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-inner-hits.html) ---
**Author:** ![TheConnMan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theconnman/32/36487_2.png) [@TheConnMan](https://discuss.elastic.co/u/TheConnMan)\ **Post date:** [October 14, 2018, 2:27pm UTC](https://discuss.elastic.co/t/parent-child-indices-getting-out-of-sync/152281/3 "2018-10-14T14:27:02Z")
Hi Lee, Thanks for your response. I'm not sure which errors exactly _could_ be getting thrown, but when I overload the bulk indexing thread pool I do get those errors as part of the bulk response. My question is whether or not there are other thread pools getting overloaded (or something else) which is causing the the documents to not be saved, but isn't returning an error in the bulk index response. Thanks for the `inner hits` recommendation, I'll look into it more. The four core queries we make are: - Search for matching parent documents (trivial with a single index) - Search for matching nested documents (looks to be possible with the `inner_hits` query) - Aggregate matching parent documents (also trivial with a single index) - Aggregate matching nested documents (TBD) If we can search for _and_ aggregate nested documents then perhaps all my troubles are solved. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 11, 2018, 2:27pm UTC](https://discuss.elastic.co/t/parent-child-indices-getting-out-of-sync/152281/4 "2018-11-11T14:27:03Z")
Closed on November 11, 2018, 2:27pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.