Parent child boost not working consistently

Elastic Stack Elasticsearch
1

hi,

We are using ES 7.10.2 on Windows 2012 R2, using inbuilt OpenJDK packaged with the MSI. We are seeing issues happening randomly with ES query being run on index using parent child relationship.

We overwrite the records on a nightly basis via bulk API using index action on the bulk request. On few days, when we try to query the parent using a child condition, ES query does not return any data. Reloading the child records solves the problem.

But the issue is not reproducible at will. It happens randomly - once or twice in a month. In our nightly process, child records are indexed first. Parent records are indexed later.

Is there is known restriction around the order in which parent child records have to be loaded. We are assigning the doc id ourselves. So 99% of the records will continue to have same doc id, unless some brand new records were created. But the issue is seen on existing records itself. So we are not able to figure out, how to trouble shoot the issue.

The child query has a boost clause and overall search clause uses a min_score condition. At high level, we suspect that the boost from has_child query is not being applied consistently causing the ES query to not return data. But like I mentioned above, we cannot reproduce the issue at will.

If anyone has faced similar issue, can you please provide some guidance on how to troubleshoot the issue?

Thanks!

3

Any chance you can upgrade to the latest minor version and see if that problem persists? Also, how do you index your data? Are you making sure that the correct routing information is set on indexing always?

4

We index data only via bulk API. Even when the issue is occurring, we can still see the routing information properly populated on child record. We are in the processing of planning upgrade to either 7.12.0 or 7.16.3. Will either of that be a good version to test with?

5

Upgrading to the latest version to me is always worth a test.

One more thing to test: When you run into this issue, can you run a search for the _id using a term query on the index. I would like to be sure that the document is not accidentally indexed into another shard than the parent.

6

@spinscale how do I verify under which shard the document is stored?

7

Run a search on the index for the _id field, you might get back two documents. In order to see which shard they are in you can add "explain": true.

hope this helps

8

we wouldn't get 2 documents back as the _id has to be unique for both parent and child documents. We are anyway adding custom id attribute also which will return both docs. We will try to check the assigned shard using explain, whenever we encounter the issue next time.

9

@spinscale, we had reoccurrence of this issue in production that is still running 7.10.2. Using explain, we were able to confirm that both parent and child record is stored on same shard. Just that boost applied on has_child condition was NOT resulting in increase in score, causing the search to not return matching records (we have min score condition). Only additional information that we gathered was that the child record had an earlier update timestamp compared to parent record. I am not sure if this makes any difference.

10

Using the explain API you should also see score calculations. Can you see how they differ in your two cases?

11

Unfortunately, since it was prod issue, we had to reload the data to get the app working again. But based on returned score, we can easily say that the below was the one that was making the difference. When issue is present, we get some score of 500, where as we expect to get 10500.

Below is what I see, when its working properly.

              {
                  "value" : 10000.0,
                  "description" : "A match, join value 1000080000000308213",
                  "details" : [ ]
                }

Below is the has child query, where we expect to get the 10000 boost in score, which we don't get whenever the issue happens. So even though a match is found, due to min score, record will not be returned.

        {
          "has_child": {
            "type": "childrecordtype",
            "score_mode": "sum",
            "query": {
              "bool": {
                "should": [
                  {
                    "multi_match": {
                      "query": "AB123",
                      "fields": [
                        "attr1^150",
                        "attr2^140"
                      ]
                    }
                  },
                  {
                    "terms": {
                      "visiblecode": {
                        "index": "secondindex",
                        "id": "AB123",
                        "path": "visiblecode"
                      },
                      "boost": 10000
                    }
                  }
                ]
              }
            }
          }
        }
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.