Breaking change in 7.12-13: Date processor

[Dennis1 (Dennis)]

I'd like to report a major breaking change that occurred in 7.12.1 (as originally reported and described here: Date filter problems after upgrade), and which I personally experienced when I upgraded from 7.11.1 to 7.13.2.

In short, the Date processor (for Elasticsearch ingest pipelines) and date filter plugin (for Logstash) no longer accept the use of capital 'YYYY' when defining the four-digit year syntax, and instead now only functions properly with the use of lower case letters 'yyyy'.

Moreover, when a user defines a syntax using the all-caps version, no warning or error is triggered, but rather the date is continued to be processed in some bizarre way (for example, provided the date "2021-06-24", this is converted to "2021-01-04"). This aspect seems to be more of a bug than just a breaking change.

[Badger]

Can you provide a reproducible example? This

input { generator { count => 1 lines => [ '' ] } }
filter {
    mutate { add_field => { "[encodertime]" => "2021-06-03 04:51:44" } }
    date { match => [ "encodertime", "YYYY-MM-dd HH:mm:ss" ] }
}
output { stdout { codec => rubydebug { metadata => false } } }

works just fine for me in 7.13.2.

[Dennis1 (Dennis)]

Yes, but from the Elasticsearch ingest pipeline perspective (I wasn't using Logstash personally, that was just what was reported by the OP).

PUT _ingest/pipeline/test-timestamp-pipeline
{
   "processors": [
      {
         "grok": {
            "field": "message",
            "ignore_missing": true,
            "patterns": [
               "%{TIMESTAMP_ISO8601:temp.timestamp} %{ISO8601_TIMEZONE:event.timezone}( \\(%{NOTSPACE:host.name}\\))? \\[%{NOTSPACE:process.thread.name}\\] %{WORD:log.level}%{SPACE}%{NOTSPACE:log.logger} - %{GREEDYDATA:message_remaining}"
            ],
            "on_failure": [
               {
                  "append": {
                     "field": "error.message",
                     "value": "{{ _ingest.on_failure_message }}"
                  }
               }
            ]
         }
      },
      {
         "date": {
            "if": "ctx.event?.timezone != null",
            "field": "temp.timestamp",
            "target_field": "@timestamp",
            "formats": [
               "YYYY-MM-dd HH:mm:ss.SSS",
               "YYYY MM dd HH:mm:ss.SSS"
            ],
            "timezone": "{{ event.timezone }}",
            "on_failure": [
               {
                  "append": {
                     "field": "error.message",
                     "value": "{{ _ingest.on_failure_message }}"
                  }
               }
            ]
         }
      }
   ]
}

POST _ingest/pipeline/test-timestamp-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "2021-06-04 10:06:58.825 -0700 [http-nio-8080-exec-18] INFO  org.fake.test.util.Helper - Setting the scoring strategy for query with ID [ fakeid1 ] to default"
      }
    }
  ]
}

Response (see @timestamp):

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "temp" : {
            "timestamp" : "2021-06-04 10:06:58.825"
          },
          "process" : {
            "thread" : {
              "name" : "http-nio-8080-exec-18"
            }
          },
          "@timestamp" : "2021-01-04T10:06:58.825-07:00",
          "log" : {
            "level" : "INFO",
            "logger" : "org.fake.test.util.Helper"
          },
          "message_remaining" : "Setting the scoring strategy for query with ID [ fakeid1 ] to default",
          "message" : "2021-06-04 10:06:58.825 -0700 [http-nio-8080-exec-18] INFO  org.fake.test.util.Helper - Setting the scoring strategy for query with ID [ fakeid1 ] to default",
          "event" : {
            "timezone" : "-0700"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-06-25T17:38:01.508130253Z"
        }
      }
    }
  ]
}

Then, if you use yyyy instead of YYYY, it processes it properly. Strange, right?

[Badger]

If you set the day of the month to more than 31 do you get a parse error or does it move the date into February?

[Dennis1 (Dennis)]

@badger, yes if I set the day of the month to greater than 31, I get a parse error. And if I set the month to greater than 12, I get a parse error.