Basically I am not getting a @timestamp field that is a date - its coming through as text.
#this is my .conf file (with some stuff excluded for security)
input{
beats{
port => 5048
}
}
filter {
json {
source => "message"
}
date {
match => [ "timestamp", "YYYY-MM-ddTHH:mm:ss,sss+hhhh" ] }
}
output {
elasticsearch - #this is excluded as i know this works.
You mean the timestamp field, not the @timestamp right?
The @timestamp field coming from Logstash will always be a date field, even if you do not have a date filter or if the date filter fails.
date {
match => [ "timestamp", "YYYY-MM-ddTHH:mm:ss,sss+hhhh" ]
}
This filter will parse the timestamp field from the date string provided and save it as @timestamp, which is the default target, the original timestamp field will not be changed and if Elastiscearch does not recognized it as a date string, which can happen sometimes, it will be stored as a string.
If you want to make sure that the timestamp field is saved as a date in elasticsearch, you need to have another date filter where you will set it as a target.
date {
match => [ "timestamp", "YYYY-MM-ddTHH:mm:ss,sss+hhhh" ]
target => "timestamp"
}
This will convert the timestamp field from a string to a date field.
You may need to recreate your destination index as this will change the mapping of the field.
So basically, when i create the index pattern, it tells me that the @timestamp field is text as is Time. Because this is missing that index pattern isn't viewable in discover.
I don't think this will match the format of the @timestamp field generated by Logstash, that would explain why it was mapped as a text.
I would suggest that you remove this dynamic_date_formats, since this is not needed in most cases and could be seen as a more advanced setting that you do not want when you are starting using Elastic.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.