Use date field to copy year to another field index

[dlaprade]

Elasticsearch version:

6.3

Situation:

I have a date field that contains a full date:
"releaseDate" : { "type" : "date", "format" : "yyyy-MM-dd'T'HH:mm:ssZ" }
I would like to copy the year out of a that field into another field. Is this possible to do at index time?

[dlaprade]

I realize that I could use 'copy_to' to copy the "releaseDate" into another field that uses a normalizer with a char_filter to strip out the year but the "copy_to" documentation states:

You cannot copy recursively via intermediary fields

Maybe that is why this doesn't work:

 "releaseDate" : {
                "type" : "date",
                "format" : "yyyy-MM-dd'T'HH:mm:ssZ",
                "copy_to":"releaseYear"
            },
            "releaseYear" : {
            	"type": "keyword",
            	"normalizer": "year_filtered_normalizer",
            	"copy_to": "searchterms"
            },

And I tried:

"releaseDate" : {
                "type" : "date",
                "format" : "yyyy-MM-dd'T'HH:mm:ssZ",
                "fields" : {
                	"year" : {
                		"type": "keyword",
	                	"normalizer": "year_filtered_normalizer",
	                	"copy_to": "searchterms"
                	}
                }
            },

The above gives my an error on indexing:

"copy_to in multi fields is not allowed. Found the copy_to in field [year] which is within a multi field."

Is it possible to even do what I want using settings/mappings?

[gabriel_tessier]

Hi,

Did you check about ingest to reformat your date?

https://www.elastic.co/guide/en/elasticsearch/reference/current/date-processor.html

[dlaprade]

When using Elastic version 6.7, I created an ingest pipeline with a given name.

{
	"description" : "A pre-processor to handle pulling the release year out of the releaseDate field.",
    "processors": [
        {
            "date": {
                "field": "releaseDate",
                "target_field": "releaseYear",
                "formats": [
                    "yyyy"
                ],
                "ignore_failure" : true
            }
        }
    ]
}

I created a new index that set the default_pipeline to be my created pipeline.

"settings": {
    "default_pipeline": "my-pipeline",

When re-indexing the data into the new index, ingest pipeline does not fire and therefore the year field does not get populated. Am I doing something incorrectly?

[gabriel_tessier]

Hi,

without

"ignore_failure" : true

Did you have error message if the year is not set there's certainly a raised error.

[dlaprade]

In my current data set, all records have a releaseDate .

[gabriel_tessier]

but if you remove ignore_failure do you have any error message that can help?

Or better can you provide an example of document that you store, it can help to reproduce the problem.

[spinscale (Alexander Reelsen)]

How about this

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "ctx.year = ZonedDateTime.parse(ctx.foo).getYear()"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "foo": "2019-06-17T12:34:56.789Z"
      }
    }
  ]
}