Currently, the Template [1] for one of my indices contains a section like this, since LogStash is creating field "my_datefield" as a string, with one of these two patterns.
I need to change the way LogStash builds "my_datefield", to use the Filter plugin Date. Therefore, the type of "my_datefield" won't be anymore a "string" but a "LogStash::Timestamp" type. "my_datefield" will look like this
"my_datefield" => 2019-08-01T20:55:48.719Z
I assume I can create a new Template [2] to address the new type, like this
"my_datefield": {
"type": "date"
},
My understanding is that the new Template [2] only will affect new indices. Correct? As they are created daily, "my_index-yyyy.mm.dd", that means Template [2] will start being used the day after I set it up, right?
So my question is how to allow ElasticSearch to accept both types, string and timestamp, during the day of the transition. The day I set the new code for LogStash, the index for that day still uses Template [1], but "my_datefield" won't be a string anymore. My experiments tell me that ElasticSearch will reject the new documents as it does not know how parse the field "my_datefield" anymore. Is there a solution for this?
you are right regarding the template, it will only kick into effect once a new index is created. However, due to sending the date as a string, you will be able to send the date as a string on day X (and it will be indexed as a string) and as a string on day X+1 and it will be indexed as a date.
One thing that could happen however is, that documents start to get rejected, because the date is not in the required format.
The ' " ' part is to escape the single quotes around T and Z, as I upload the template entirely from the command line.
So, will adding yyyy-MM-dd'T'HH:mm:ss.SSS'Z' to the format make ElasticSearch to accept documents when date fields become Logstash::timestamp instead of string?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.