Preventing date_time_parse_exception in ingest pipeline

Elastic Stack Elasticsearch
,
1

Hi, I have encountered date parsing related problem in my ingest pipeline. Full details can be found here: [Ingest pipeline] Occasionally ends with `date_time_parse_exception` when pipeline is trying to store time without seconds part · Issue #97314 · elastic/elasticsearch · GitHub . However the main problem is that ingest pipeline with configuration like this (configured via Nest):

 new SetProcessor
 {
     Field = new Field("ingest_timestamp"),
     Description = "Generic pipeline tracking ingest timestamp.",
     Value = "{{_ingest.timestamp}}"
 }

can lead to errors because {{_ingest.timestamp}} relies on Mustache to-String-ing which can omit zeros when date is for example 2023-06-02T13:57:00.000Z. When such value is being inserted into property with given mapping:

 "ingest_timestamp": {
        "type": "date",
        "format": "strict_date_optional_time_nanos"
      },

it will fail. In the mentioned issue there is a proposed fix for that but I am unable to configure it via Nest. When below code is executed:

var putResponse = await elasticClient.Ingest.PutPipelineAsync(pipelineName, p => p
    .Processors(pr => pr
        .Script(s => s
            .Source("ctx.ingest_timestamp = metadata().now.format(DateTimeFormatter.ISO_INSTANT);")
        )
    ), cancellationToken);

it ends up with: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: PUT /_ingest/pipeline/my-pipeline-name?pretty=true. ServerError: Type: script_exception Reason: "compile error" CausedBy: "Type: illegal_argument_exception Reason: "Unknown call [metadata] with [0] arguments.""

Can someone suggest how to configure it using Nest?

Elasticsearch version: 7.17.25 (but soon we will be updating to 8.X where this pipeline will still be required)
Nest version: 7.17.5

2

The Nest client has nothing to do with it. The metadata() call doesn’t exist in 7.x, it was added three years ago in v8.4.0.

3 
POST _ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "ingest_timestamp",
          "value": "{{_ingest.timestamp}}"
        }
      },
      {
        "date": {
          "field": "ingest_timestamp",
          "formats": ["strict_date_optional_time",
                      "strict_date_optional_time_nanos"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
      }
    }
  ]
}

This seems to work for me on 7.17.25, but I haven’t tested it especially rigorously. Another approach would be to grab the current time with a set processor like I’ve done here and then to manipulate that using data parsing and formatting in a script processor.

4

If in your example ingest_timestamp is of type strict_date_optional_time_nanos and this execution will happen at time without seconds (e.g. 12:25:00) would’t it fail like in mentioned Github issue? I am asking because this set processor looks exactly like the one I was using before.

5

Yes, the set processor is the same. The date processor that I’m using here parses the date with two different formats, and emits a date in ISO-8601 format. So, no, it wouldn’t fail like mentioned in the github issue.

How about you try it a few dozen thousand times and see?

6

Thanks, so if I will execute request like this

POST _ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "ingest_timestamp",
          "value": "2023-06-02T13:57Z" //Hardcoded date without seconds
        }
      },
      {
        "date": {
          "field": "ingest_timestamp",
          "formats": ["strict_date_optional_time",
                      "strict_date_optional_time_nanos"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
      }
    }
  ]
}

This should produce date in ISO-8601 format on document right? I am asking because in response I can see:

"doc" : {
    "_index" : "index",
    "_type" : "_doc",
    "_id" : "id",
    "_source" : {
        "ingest_timestamp" : "2023-06-02T13:57Z"
    },
    "_ingest" : {
        "pipeline" : "_simulate_pipeline",
        "timestamp" : "2025-10-16T07:48:33.376594276Z"
    }
}