Completion suggestor field returns both old and updated document results

[Cara_B]

Hi All,

We've noticed that on updating a document completion field that the results will then return both the old document completion field and the updated version. If I update again we continue to get 2 results - the original and the latest updated value.

We're using Nest, populating an object to be indexed.

The suggest field mapping:

 "suggestField": {
        "type": "completion",
        "analyzer": "standard",
        "payloads": true,
        "context": {
          "lang": {
            "type": "category",
            "default": [ "en" ]
          },
          "stock": {
            "type": "category",
            "default": [ "GPS" ]
          }
        }
      }

But I see the issue when I update this document and query it:
Example Sense query:

  GET savills-alias/elasticlocationsynonyms/_search
  {
  "size":0,"suggest":{"textsuggest":{"text":"sw9","completion":{"context":   
 {"lang":"en","stock":"GRS_PT"},"field":"suggestField","size":10}}}
 }

I get more than one result - the first indexed plus the latest updated.

Output:

---

{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1032041,
"max_score": 0,
"hits": []
},
"suggest": {
"textsuggest": [
{
"text": "sw9",
"offset": 0,
"length": 3,
"options": [
{
"text": "SW9, England",
"score": 0,
"payload": {
"locationId": "45756",
"autoCompleteKey": "SW9, England",
"translatedAutoCompleteKey": "SW9, England",
"language": "en",
"population": 0,
"country": "England",
"flagFileName": "flag-england.png",
"synonymId": 2827453
}
},
{
"text": "SW9, England-TEST3",
"score": 0,
"payload": {
"locationId": "45756",
"autoCompleteKey": "SW9, England-TEST3",
"translatedAutoCompleteKey": "SW9, England-TEST3",
"language": "en",
"population": 0,
"country": "England",
"flagFileName": "flag-england.png",
"synonymId": 2827453
}
}
]
}
]
}
}

---

The synonymId in both results is the documentId so it isn't being duplicated.

Any ideas? I see there have been some issues with the completion field and deletes in the past. Is this potentially something along the same lines?

ES version 2.3.0. Let me know if you need anymore details.

Thanks
Cara

[danielmitterdorfer]

Hi @Cara_B,

based on your description it is highly likely that you see this issue here. You will see the old document in suggestions until the segment is merged and thus the document is physically deleted. Quoting the reference docs:

The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that.

(note: "optimize" is now called "force-merge").

Daniel

[Cara_B]

Hi @danielmitterdorfer,

Thanks for the response. This definitely sounds like the issue. We're using ES 2.3.0 and I'm weary of running a force-merge/optimize after reading of some of the impacts it may have on search. We're happy to stop indexing while this takes place but the search queries must not be impacted. Do you know of any other options?

Thanks
Cara

[danielmitterdorfer]

Hi @Cara_B,

another option would be an upgrade to Elasticsearch 5 where completion suggester does not return deleted documents anymore.

Daniel