Documents doesn't get updated after the version is updated & response is successful in ES

Elastic Stack Elasticsearch
1

Hi ,
I am facing an issue on AWS Elastic search instance.
I am sending POST api request to create/update the document in ES , somehow the version gets updated but on querying the record , it still shows the old record.

I am getting the below response :Response:
{
"_index": "test",
"_type": "_doc",
"_id": "22492123",
"_version": 4,
"result": "updated",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 197,
"_primary_term": 1
}

Can help to guide on the same.

Thanks

2

How did you retrieve the document you showed? Did you use the GET API? If not, did you ensure a refresh was performed before retrieving the document?

3

Hi Christian,

This response I get when I am making POST call through API , something like
POST /test-index/_doc/920

Currently we need to update the status field in each document . We are making multiple calls to update documents but somehow they are not reflecting in ES although from logs I can see response is successful.

I added ?refresh=true but still same.

4

The response says the document was updated. How do you determine it was not? What query do you run?

5

Yes the response say its updated but when I query

GET /test-index/_doc/920
{
"from": 0,
"size": 400,
"query": {
"match_all": {}
}
The version is updated but the status (field inside document) is old.

6

Can you please provide the exact steps required to recreate this issue, e.g. through Kibana dev tools? This would allow us to test and troubleshoot this against a standard Elasticsearch cluster. AWS Elasticsearch are running a forked version so we need to reproduce this on standard Elasticsearch in order to be able to help.

1 Like
7

Sure .

Currently we have records in Dynamodb which we are streaming into ElasticSearch through Lambda .
Any new or modified records in db triggers lambda which makes POST call to Elastic Search .

Below is the snippet for Post Request -

request.path = path.join('/',esDomain.index,esDomain.doctype,id);
request.body = JSON.stringify(doc);
request.headers['Content-Length'] = Buffer.byteLength(request.body);
 request.method = 'POST';
request.path = path.join('/',esDomain.index,esDomain.doctype,id+'?refresh=true');
console.log("API call",request.path)
request.body = JSON.stringify(doc);
request.headers['Content-Length'] = Buffer.byteLength(request.body);
const send = new AWS.HttpClient();
setTimeout(function () {
    send.handleRequest(request, null, function(httpResp) {
    var respBody = '';
    httpResp.on('data', function (chunk) {
        respBody += chunk;
    });
    httpResp.on('end', function (chunk) {
        console.log('Response: ' + respBody);
        context.succeed('Lambda added document ' + doc);
    });
}, function(err) {
    console.log('Error: ' + err);
    context.fail('Lambda failed with error ' + err);
});},3000)

}

I tried adding timeout as well so that when record is updated, version shouldn't conflict but no luck.

8

Also I tested through Kibana Dev Tools too but i see the same record.

9

Can you please provide an example we can run through Kibana dev console that replicates the issue?

10

The below query when I executed on Kibana dev tools -

GET /test-index/_doc/920
{
"from": 0,
"size": 400,
"query": {
"match_all": {}
}

I got below response -

{
"_index" : "test-index",
"_type" : "_doc",
"_id" : "920",
"_version" : 4,
"_seq_no" : 206,
"_primary_term" : 1,
"found" : true,
"_source" : {
"Status" : "Processing",
}
}

So the version is 4 here but status is still old .

As per logs the status should be "Finished"

11

Please show all the steps/requests, e.g.

  1. Insert initial document
  2. Update document
  3. Run refresh
  4. Get updated document and check status
12

As mentioned above , I am adding through Post API so this happens through this flow .
If i add or update through Dev Tools , it works .

Only via Api , document is not getting updated . Creating of new documents works fine.

From Kibana Dev Tools , the same works -

POST /test-index/_doc/1?refresh=true
{
"DataSourceStatus" : "Processing"
}

POST /test-index/_doc/1?refresh=true
{
"DataSourceStatus" : "Processing_1"
}

GET /test-index/_doc/1 gives correct result-
{
"_index" : "test-index-000001",
"_type" : "_doc",
"_id" : "1",
"_version" : 3,
"_seq_no" : 2,
"_primary_term" : 1,
"found" : true,
"_source" : {
"DataSourceStatus" : "Processing_1"
}
}

13

Are you sure your update process generates the requests you expect? If you are not able to reproduce it outside your application I would suspect there is something wrong there.

15

Currently I am using the same Post request (POST /test-index/_doc/1?refresh=true) for both Updating & Creating the record . Will that make any difference.

Also we are updating same record multiple time within seconds

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.