Preventing date_time_parse_exception in ingest pipeline

Elastic Stack Elasticsearch
,
1

Hi, I have encountered date parsing related problem in my ingest pipeline. Full details can be found here: [Ingest pipeline] Occasionally ends with `date_time_parse_exception` when pipeline is trying to store time without seconds part · Issue #97314 · elastic/elasticsearch · GitHub . However the main problem is that ingest pipeline with configuration like this (configured via Nest):

 new SetProcessor
 {
     Field = new Field("ingest_timestamp"),
     Description = "Generic pipeline tracking ingest timestamp.",
     Value = "{{_ingest.timestamp}}"
 }

can lead to errors because {{_ingest.timestamp}} relies on Mustache to-String-ing which can omit zeros when date is for example 2023-06-02T13:57:00.000Z. When such value is being inserted into property with given mapping:

 "ingest_timestamp": {
        "type": "date",
        "format": "strict_date_optional_time_nanos"
      },

it will fail. In the mentioned issue there is a proposed fix for that but I am unable to configure it via Nest. When below code is executed:

var putResponse = await elasticClient.Ingest.PutPipelineAsync(pipelineName, p => p
    .Processors(pr => pr
        .Script(s => s
            .Source("ctx.ingest_timestamp = metadata().now.format(DateTimeFormatter.ISO_INSTANT);")
        )
    ), cancellationToken);

it ends up with: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: PUT /_ingest/pipeline/my-pipeline-name?pretty=true. ServerError: Type: script_exception Reason: "compile error" CausedBy: "Type: illegal_argument_exception Reason: "Unknown call [metadata] with [0] arguments.""

Can someone suggest how to configure it using Nest?

Elasticsearch version: 7.17.25 (but soon we will be updating to 8.X where this pipeline will still be required)
Nest version: 7.17.5

2

The Nest client has nothing to do with it. The metadata() call doesn’t exist in 7.x, it was added three years ago in v8.4.0.

3 
POST _ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "ingest_timestamp",
          "value": "{{_ingest.timestamp}}"
        }
      },
      {
        "date": {
          "field": "ingest_timestamp",
          "formats": ["strict_date_optional_time",
                      "strict_date_optional_time_nanos"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
      }
    }
  ]
}

This seems to work for me on 7.17.25, but I haven’t tested it especially rigorously. Another approach would be to grab the current time with a set processor like I’ve done here and then to manipulate that using data parsing and formatting in a script processor.

4

If in your example ingest_timestamp is of type strict_date_optional_time_nanos and this execution will happen at time without seconds (e.g. 12:25:00) would’t it fail like in mentioned Github issue? I am asking because this set processor looks exactly like the one I was using before.

5

Yes, the set processor is the same. The date processor that I’m using here parses the date with two different formats, and emits a date in ISO-8601 format. So, no, it wouldn’t fail like mentioned in the github issue.

How about you try it a few dozen thousand times and see?

6

Thanks, so if I will execute request like this

POST _ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "ingest_timestamp",
          "value": "2023-06-02T13:57Z" //Hardcoded date without seconds
        }
      },
      {
        "date": {
          "field": "ingest_timestamp",
          "formats": ["strict_date_optional_time",
                      "strict_date_optional_time_nanos"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
      }
    }
  ]
}

This should produce date in ISO-8601 format on document right? I am asking because in response I can see:

"doc" : {
    "_index" : "index",
    "_type" : "_doc",
    "_id" : "id",
    "_source" : {
        "ingest_timestamp" : "2023-06-02T13:57Z"
    },
    "_ingest" : {
        "pipeline" : "_simulate_pipeline",
        "timestamp" : "2025-10-16T07:48:33.376594276Z"
    }
}
7

I tried this pipeline:

PUT _ingest/pipeline/timestamp-demo
{
  "processors": [
    {
      "set": {
        "field": "ingest_timestamp",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "date": {
        "field": "ingest_timestamp",
        "target_field": "ingest_timestamp",
        "formats": [
          "strict_date_optional_time",
          "strict_date_optional_time_nanos"
        ],
        "output_format": "strict_date_optional_time_nanos"
      }
    }
  ]
}

With this mapping:

PUT my-index
{
  "mappings": {
    "properties": {
      "ingest_timestamp": {
        "type":   "date",
        "format": "strict_date_optional_time_nanos"
      }
    }
  }
}

And with this change to Elasticsearch itself in order to make interesting timestamps more frequent (for testing purposes):

diff --git a/server/src/main/java/org/elasticsearch/ingest/IngestDocument.java b/server/src/main/java/org/elasticsearch/ingest/IngestDocument.java
index d0d7687fcbde..39ccc4f4842f 100644
--- a/server/src/main/java/org/elasticsearch/ingest/IngestDocument.java
+++ b/server/src/main/java/org/elasticsearch/ingest/IngestDocument.java
@@ -44,6 +44,7 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.BiConsumer;
 import java.util.stream.Collectors;
 
@@ -101,8 +102,18 @@ public final class IngestDocument {
     private boolean reroute = false;
     private boolean terminate = false;
 
+    private static final AtomicInteger counter = new AtomicInteger(-1);
+
     public IngestDocument(String index, String id, long version, String routing, VersionType versionType, Map<String, Object> source) {
-        this.ctxMap = new IngestCtxMap(index, id, version, routing, versionType, ZonedDateTime.now(ZoneOffset.UTC), source);
+        final var now = switch (counter.incrementAndGet() % 4) {
+            case 0 -> ZonedDateTime.now(ZoneOffset.UTC).withNano(0).withSecond(0);
+            case 1 -> ZonedDateTime.now(ZoneOffset.UTC).withNano(0);
+            case 2 -> ZonedDateTime.now(ZoneOffset.UTC).withNano(123_000_000);
+            case 3 -> ZonedDateTime.now(ZoneOffset.UTC);
+            default -> throw new IllegalStateException();
+        };
+
+        this.ctxMap = new IngestCtxMap(index, id, version, routing, versionType, now, source);
         this.ingestMetadata = new HashMap<>();
         this.ingestMetadata.put(TIMESTAMP, ctxMap.getMetadata().getNow());
         this.templateModel = initializeTemplateModel();

And I think it seems like it’s all working fine to me. In my previous pipeline I missed that the target_field and output_format are probably important.