_update_by_query "lenient" flag not recognized

Elastic Stack Elasticsearch
1

I'm having an issue with updating an index via _update_by_query. I have certain fields that are causing format errors so I'm trying to skip them using the "lenient" flag outlined in the documentation. I'm using elasticsearch 7.10.

Query:

    POST /dummy-index/_update_by_query?lenient=true
    {
      "query": { 
        "term": {
          "user.id": "kimchy"
        }
      }
    }

Result:

    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "request [/dummy-index/_update_by_query] contains unrecognized parameter: [lenient]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "request [/dummy-index/_update_by_query] contains unrecognized parameter: [lenient]"
      },
      "status" : 400
    }

To give a bit more context this is the error I face when I run the update query:

          "cause" : {
            "type" : "mapper_parsing_exception",
            "reason" : "failed to parse field [@timestamp] of type [date] in document with id 'lZCFKncBPKE3DmYvI5D0'. Preview of field's value: '1.590399287522615E9'",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [1.590399287522615E9] with format [epoch_second||date_time]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Failed to parse with all enclosed parsers"
              }
            }
          }

caused by some old documents that had a misconfigured timestamp field. Incidentally this doesn't seem to cause any issues when doing a re-index operation against the same index template.

Am I able to somehow skip the errors and only update the documents with valid timestamps? Alterantively, can I filter out the offending documents and remove them from the index?

Appreciate any help!

2

I managed to get it working for my use case. Since certain timestamps were stored as a double I had to convert them during the update. Something like:

if(ctx._source['@timestamp'] instanceof double){
    ctx._source['@timestamp'] = new Date( ( (long) ctx._source['@timestamp'] ) * 1000);
}

I still think that there is some sort of bug with Elastic not recognizing the "lenient" flag and I'm not sure why reindexing doesn't cause these errors.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.