How to deal with the transition when a field changes format "string" -> "timestamp"

Elastic Stack Elasticsearch
1

Hello.

Facts:

  • Using ElasticSearch version 5.6.10

  • Currently, the Template [1] for one of my indices contains a section like this, since LogStash is creating field "my_datefield" as a string, with one of these two patterns.

     "my_datefield": {
      "type": "date",
       "format": "MM/dd/yy HH:mm:ss||MM/dd/yy HH:mm:ss.SSS"
},

  • I need to change the way LogStash builds "my_datefield", to use the Filter plugin Date. Therefore, the type of "my_datefield" won't be anymore a "string" but a "LogStash::Timestamp" type. "my_datefield" will look like this

    "my_datefield" => 2019-08-01T20:55:48.719Z

  • I assume I can create a new Template [2] to address the new type, like this

     "my_datefield": {
      "type": "date"
},

  • My understanding is that the new Template [2] only will affect new indices. Correct? As they are created daily, "my_index-yyyy.mm.dd", that means Template [2] will start being used the day after I set it up, right?

So my question is how to allow ElasticSearch to accept both types, string and timestamp, during the day of the transition. The day I set the new code for LogStash, the index for that day still uses Template [1], but "my_datefield" won't be a string anymore. My experiments tell me that ElasticSearch will reject the new documents as it does not know how parse the field "my_datefield" anymore. Is there a solution for this?

Thanks a lot in advance.

2

you are right regarding the template, it will only kick into effect once a new index is created. However, due to sending the date as a string, you will be able to send the date as a string on day X (and it will be indexed as a string) and as a string on day X+1 and it will be indexed as a date.

One thing that could happen however is, that documents start to get rejected, because the date is not in the required format.

3

I have changed the Template, adding a new format to the property, like this

"my_datefield": {
    "type": "date",
    "format": "MM/dd/yy HH:mm:ss||MM/dd/yy HH:mm:ss.SSS||MM/dd/yy HH:mm:ss.1000||yyyy-MM-dd'"'T'"'HH:mm:ss.SSS'"'Z'"'"
}

The ' " ' part is to escape the single quotes around T and Z, as I upload the template entirely from the command line.

So, will adding yyyy-MM-dd'T'HH:mm:ss.SSS'Z' to the format make ElasticSearch to accept documents when date fields become Logstash::timestamp instead of string?

4

there is no such thing as a logstash timestamp for Elasticsearch. A value is received as a string and tried to be converted.

--Alex

5

yeah, you are right.
Adding "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" did the trick :slight_smile: