DateFormat in ingest pipeline

[Mukesh_Rawat]

Hi i have created ingest pipeline to fetch custom logs, my pipeline with processor looks like below

[
  {
    "grok": {
      "field": "message",
      "patterns": [
        "\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{DATA:env}\\.%{DATA:log.level}: (?<message>(.|\r|\n)*)"
      ],
      "ignore_missing": true
    }
  },
  {
    "date": {
      "field": "timestamp",
      "formats": [
        "yyyy-MM-dd'T'HH:mm:ss.SSXX"
      ],
      "target_field": "@timestamp"
    }
  },
  {
    "json": {
      "field": "message",
      "add_to_root": true,
      "ignore_failure": true
    }
  }
]

now when i am sending logs in date format [2022-01-27T08:31:16.806171+00:00]

it gives error
{"type":"illegal_argument_exception","reason":"failed to parse date field [2022-01-27T10:22:49.234717+00:00] with format [yyyy-MM-dd'T'HH:mm:ss.SSXX]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2022-01-27T10:22:49.234717+00:00' could not be parsed at index 22"}}

can anyone help what exactly wrong here

[Tomo_M (Tomohiro Mitani)]

@Mukesh_Rawat

[2022-01-27T08:31:16.806171+00:00]

The format is wrong. The format should be "yyyy-MM-dd'T'HH:mm:ss.SSSSSSXXX".
See here and here carefully.

You can use this value also in date field but the microsecond accuracy will be discarded. To hold the accuracy use date_nanos field type.

Fraction : Outputs the nano-of-second field as a fraction-of-second. The nano-of-second value has nine digits, thus the count of pattern letters is from 1 to 9. If it is less than 9, then the nano-of-second value is truncated, with only the most significant digits being output.

Offset X and x :Three letters outputs the hour and minute, with a colon, such as '+01:30'.