Data_detection setting in the ECS index template

[Chris_Stone]

Hi, I posted first in the Logstash category, and it looks like this might be more Elasticsearch related...

I am collecting http access logs and wanting to place some date headers into date type fields. I've tried a couple of things in Logstash, shown below, but while the date format is converted properly, the resulting field is still of type keyword rather than date.

To be clear, I'm receiving RFC 1123 formatted dates and they are being converted to the Elasticsearch date format (right?):

Wed, 25 Nov 2020 19:24:27 GMT ---> 2020-11-25T19:24:27.000Z

But the resulting field is still a string type.

date{
  match => ["[respHdr][date]", "EEE, dd MMM yyyy HH:mm:ss zzz"]
  target => "[http][response][date]"
  remove_field => [ "[respHdr][date]" ]
}

date{
  match => ["[http][response][expires]", "EEE, dd MMM yyyy HH:mm:ss zzz"]
  target => "[http][response][expires]"
}

I do see that placing a date into @timestamp (the default target) works fine:

   date{
    match => ["start", "UNIX"]
  }

I have deleted the index and started from fresh each time, but am seeing the same behavior. I'm using the ecs-logstash index template and ELK 7.10.

I see that date_detection is false by default in the ECS template and it was suggested that this could be the issue. If so, what are my options going forward?

1. Set data_detection to true? What would be the ramifications of this, and what would be the best way to do it.

2. Add the new date fields to the index template explicitly? Again, What would be the ramifications of this, and what would be the best way to do it.

I'd appreciate any pointers on getting this to work and am happy to provide any further needed information.

Thanks,

--Chris

[Chris_Stone]

Well, I'll go ahead and edit the ecs-logstash template, set date_detection to true and see what happens...