Preventing date_time_parse_exception in ingest pipeline

marekott
Tue, 14 Oct 2025 13:30:36 +0000
Elasticsearch

Hi, I have encountered date parsing related problem in my ingest pipeline. Full details can be found here: [Ingest pipeline] Occasionally ends with `date_time_parse_exception` when pipeline is trying to store time without seconds part · Issue #97314 · elastic/elasticsearch · GitHub . However the main problem is that ingest pipeline with configuration like this (configured via Nest):

 new SetProcessor
 {
     Field = new Field("ingest_timestamp"),
     Description = "Generic pipeline tracking ingest timestamp.",
     Value = "{{_ingest.timestamp}}"
 }

can lead to errors because {{_ingest.timestamp}} relies on Mustache to-String-ing which can omit zeros when date is for example 2023-06-02T13:57:00.000Z. When such value is being inserted into property with given mapping:

 "ingest_timestamp": {
        "type": "date",
        "format": "strict_date_optional_time_nanos"
      },

it will fail. In the mentioned issue there is a proposed fix for that but I am unable to configure it via Nest. When below code is executed:

var putResponse = await elasticClient.Ingest.PutPipelineAsync(pipelineName, p => p
    .Processors(pr => pr
        .Script(s => s
            .Source("ctx.ingest_timestamp = metadata().now.format(DateTimeFormatter.ISO_INSTANT);")
        )
    ), cancellationToken);

it ends up with: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: PUT /_ingest/pipeline/my-pipeline-name?pretty=true. ServerError: Type: script_exception Reason: "compile error" CausedBy: "Type: illegal_argument_exception Reason: "Unknown call [metadata] with [0] arguments.""

Can someone suggest how to configure it using Nest?

Elasticsearch version: 7.17.25 (but soon we will be updating to 8.X where this pipeline will still be required)
Nest version: 7.17.5

7 posts - 2 participants