_update_by_query "lenient" flag not recognized

I'm having an issue with updating an index via _update_by_query. I have certain fields that are causing format errors so I'm trying to skip them using the "lenient" flag outlined in the documentation. I'm using elasticsearch 7.10.

Query:

    POST /dummy-index/_update_by_query?lenient=true
    {
      "query": { 
        "term": {
          "user.id": "kimchy"
        }
      }
    }

Result:

    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "request [/dummy-index/_update_by_query] contains unrecognized parameter: [lenient]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "request [/dummy-index/_update_by_query] contains unrecognized parameter: [lenient]"
      },
      "status" : 400
    }

To give a bit more context this is the error I face when I run the update query:

          "cause" : {
            "type" : "mapper_parsing_exception",
            "reason" : "failed to parse field [@timestamp] of type [date] in document with id 'lZCFKncBPKE3DmYvI5D0'. Preview of field's value: '1.590399287522615E9'",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "failed to parse date field [1.590399287522615E9] with format [epoch_second||date_time]",
              "caused_by" : {
                "type" : "date_time_parse_exception",
                "reason" : "Failed to parse with all enclosed parsers"
              }
            }
          }

caused by some old documents that had a misconfigured timestamp field. Incidentally this doesn't seem to cause any issues when doing a re-index operation against the same index template.

Am I able to somehow skip the errors and only update the documents with valid timestamps? Alterantively, can I filter out the offending documents and remove them from the index?

Appreciate any help!

I managed to get it working for my use case. Since certain timestamps were stored as a double I had to convert them during the update. Something like:

if(ctx._source['@timestamp'] instanceof double){
    ctx._source['@timestamp'] = new Date( ( (long) ctx._source['@timestamp'] ) * 1000);
}

I still think that there is some sort of bug with Elastic not recognizing the "lenient" flag and I'm not sure why reindexing doesn't cause these errors.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.