Documents doesn't get updated after the version is updated & response is successful in ES

[Farah_Zaidi]

Hi ,
I am facing an issue on AWS Elastic search instance.
I am sending POST api request to create/update the document in ES , somehow the version gets updated but on querying the record , it still shows the old record.

I am getting the below response :Response:
{
"_index": "test",
"_type": "_doc",
"_id": "22492123",
"_version": 4,
"result": "updated",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 197,
"_primary_term": 1
}

Can help to guide on the same.

Thanks

[Christian_Dahlqvist]

How did you retrieve the document you showed? Did you use the GET API? If not, did you ensure a refresh was performed before retrieving the document?

[Farah_Zaidi]

Hi Christian,

This response I get when I am making POST call through API , something like
POST /test-index/_doc/920

Currently we need to update the status field in each document . We are making multiple calls to update documents but somehow they are not reflecting in ES although from logs I can see response is successful.

I added ?refresh=true but still same.

[Christian_Dahlqvist]

The response says the document was updated. How do you determine it was not? What query do you run?

[Farah_Zaidi]

Yes the response say its updated but when I query

GET /test-index/_doc/920
{
"from": 0,
"size": 400,
"query": {
"match_all": {}
}
The version is updated but the status (field inside document) is old.

[Christian_Dahlqvist]

Can you please provide the exact steps required to recreate this issue, e.g. through Kibana dev tools? This would allow us to test and troubleshoot this against a standard Elasticsearch cluster. AWS Elasticsearch are running a forked version so we need to reproduce this on standard Elasticsearch in order to be able to help.

[Farah_Zaidi]

Sure .

Currently we have records in Dynamodb which we are streaming into ElasticSearch through Lambda .
Any new or modified records in db triggers lambda which makes POST call to Elastic Search .

Below is the snippet for Post Request -

request.path = path.join('/',esDomain.index,esDomain.doctype,id);
request.body = JSON.stringify(doc);
request.headers['Content-Length'] = Buffer.byteLength(request.body);
 request.method = 'POST';
request.path = path.join('/',esDomain.index,esDomain.doctype,id+'?refresh=true');
console.log("API call",request.path)
request.body = JSON.stringify(doc);
request.headers['Content-Length'] = Buffer.byteLength(request.body);
const send = new AWS.HttpClient();
setTimeout(function () {
    send.handleRequest(request, null, function(httpResp) {
    var respBody = '';
    httpResp.on('data', function (chunk) {
        respBody += chunk;
    });
    httpResp.on('end', function (chunk) {
        console.log('Response: ' + respBody);
        context.succeed('Lambda added document ' + doc);
    });
}, function(err) {
    console.log('Error: ' + err);
    context.fail('Lambda failed with error ' + err);
});},3000)

}

I tried adding timeout as well so that when record is updated, version shouldn't conflict but no luck.

[Farah_Zaidi]

Also I tested through Kibana Dev Tools too but i see the same record.

[Christian_Dahlqvist]

Can you please provide an example we can run through Kibana dev console that replicates the issue?

[Farah_Zaidi]

The below query when I executed on Kibana dev tools -

GET /test-index/_doc/920
{
"from": 0,
"size": 400,
"query": {
"match_all": {}
}

I got below response -

{
"_index" : "test-index",
"_type" : "_doc",
"_id" : "920",
"_version" : 4,
"_seq_no" : 206,
"_primary_term" : 1,
"found" : true,
"_source" : {
"Status" : "Processing",
}
}

So the version is 4 here but status is still old .

As per logs the status should be "Finished"

[Christian_Dahlqvist]

Please show all the steps/requests, e.g.

1. Insert initial document
2. Update document
3. Run refresh
4. Get updated document and check status

[Farah_Zaidi]

As mentioned above , I am adding through Post API so this happens through this flow .
If i add or update through Dev Tools , it works .

Only via Api , document is not getting updated . Creating of new documents works fine.

From Kibana Dev Tools , the same works -

POST /test-index/_doc/1?refresh=true
{
"DataSourceStatus" : "Processing"
}

POST /test-index/_doc/1?refresh=true
{
"DataSourceStatus" : "Processing_1"
}

GET /test-index/_doc/1 gives correct result-
{
"_index" : "test-index-000001",
"_type" : "_doc",
"_id" : "1",
"_version" : 3,
"_seq_no" : 2,
"_primary_term" : 1,
"found" : true,
"_source" : {
"DataSourceStatus" : "Processing_1"
}
}

[Christian_Dahlqvist]

Are you sure your update process generates the requests you expect? If you are not able to reproduce it outside your application I would suspect there is something wrong there.

[Farah_Zaidi]

Currently I am using the same Post request (POST /test-index/_doc/1?refresh=true) for both Updating & Creating the record . Will that make any difference.

Also we are updating same record multiple time within seconds