Elastic joda to java time update - Illegal pattern component: uuuu

I'm trying update joda time to java time in pipeline.
And after copy sample from
java-time-migration-update-ingest-pipelines
I get "Illegal pattern component: uuuu" error.
Some examples

working, but changed index_name_format

also upgrade assistant - my_pipeline, fields: [[field: index_name_format, format: yyyy-w, suggestion: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.]]

reqests

PUT /_ingest/pipeline/my_pipeline?pretty
{
  "description": "Pipeline for routing data to specific index",
  "processors": [
    {
      "date": {
        "field": "createdTime",
        "formats": [
         "8uuuu-w"
        ]
      },
      "date_index_name": {
        "field": "@timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "yyyy-w"
      }
    }
  ]
}
POST /notexistingindice/doc?pipeline=my_pipeline
{
  "createdTime": "2020-15"
}
GET x-*

response

# PUT /_ingest/pipeline/my_pipeline?pretty
#! Deprecation: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.; 'Z' time zone offset/id fails when parsing 'Z' for Zulu timezone. Consider using 'X'. Prefix your date format with '8' to use the new specifier.
{
  "acknowledged" : true
}


# POST /notexistingindice/doc?pipeline=my_pipeline
#! Deprecation: the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template
#! Deprecation: 'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.
{
  "_index" : "x-2020-1",
  "_type" : "doc",
  "_id" : "ZCg4zXIB127-7IpTKDKa",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}


# GET x-*
#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
{
  "x-2020-1" : {
    "aliases" : { },
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "createdTime" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1592580909116",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "IWwAFih-QAWoatz8RnBKBw",
        "version" : {
          "created" : "6081099"
        },
        "provided_name" : "<x-{2020-1||/d{yyyy-w|UTC}}>"
      }
    }
  }
}

not working, copied from page

reqests

PUT /_ingest/pipeline/my_pipeline?pretty
{
  "description": "Pipeline for routing data to specific index",
  "processors": [
    {
      "date": {
        "field": "createdTime",
        "formats": [
         "8uuuu-w"
        ]
      },
      "date_index_name": {
        "field": "@timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "8uuuu-w"
      }
    }
  ]
}
POST /notexistingindice/doc?pipeline=my_pipeline
{
  "createdTime": "2020-20"
}
GET x-*

response

# PUT /_ingest/pipeline/my_pipeline?pretty
#! Deprecation: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.; 'Z' time zone offset/id fails when parsing 'Z' for Zulu timezone. Consider using 'X'. Prefix your date format with '8' to use the new specifier.
{
  "acknowledged" : true
}


# POST /notexistingindice/doc?pipeline=my_pipeline
{
  "error": {
"root_cause": [
  {
    "type": "illegal_argument_exception",
    "reason": "Illegal pattern component: uuuu"
  }
],
"type": "illegal_argument_exception",
"reason": "Illegal pattern component: uuuu"
  },
  "status": 400
}

but simulate is working

reqests

POST _ingest/pipeline/my_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "createdTime": "2020-20"
      }
    }
  ]
}

response

{
  "docs" : [
    {
      "doc" : {
        "_index" : "<x-{2020-1||/d{8uuuu-w|UTC}}>",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "createdTime" : "2020-20",
          "@timestamp" : "2020-01-01T00:00:00.000Z"
        },
        "_ingest" : {
          "timestamp" : "2020-06-19T15:41:57.053Z"
        }
      }
    }
  ]
}

Any suggestions?

@krrz can you confirm what version are you using?

I use 6.8.

thank you for raising this. This indeed highlights multiple problems.
Please bear in mind that you are using incorrect date formats for week based years. Joda is fairly flexible when it comes to this, but it is an undocumented implementation details that yyyy-w would allow parsing week based year. You should be using xxxx-w for joda and YYYY-w for java.time.
See https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
and https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

Apart from this I noticed these problems on ES side.
Firstly - date processor is not correctly parsing week based dates. I raised this https://github.com/elastic/elasticsearch/issues/58479

Secondly - There is also a problem with bwc layer for date index name processor. I gathered all the details here https://github.com/elastic/elasticsearch/issues/58481

Lastly - There is a problem simply parsing dates in week based dates (no ingest processor used) affecting versions 6.8 - 7.5 https://github.com/elastic/elasticsearch/issues/42588#issuecomment-648708791

Please do follow these 3 issuses and I will provide more updates there.

Thank you

Thank you for response.

From my point of view the most interesting is https://github.com/elastic/elasticsearch/issues/58481.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.