Parse timestamp not being indexed

I am attempting to parse log data using GROK and KV filters and I am running into an issue with the date\timestamp field. When I add the timestamp to my GROK statement, it is picked up and added fine when being sent to ES and I can see the timestamp field fine however I am unable to index based off of that field and it is only coming through as a string field. When I remove that GROK statement for the timestamp and instead add it into my filter, it just doesn't work or get sent over properly. Given the little information I have on the log I am working with and trying various different options, is there anything else I can try to see how I can get this working? Essentially, I want to create my index off of this timestamp field and not based off of the @timestamp default field ES provides. My config and log information is below. Thanks.

(Again, I have tried with both having timestamp in my grok and removing it and have tried different varities within my date plugin below with adding ISO, removing target, etc) The log is also above the config-I need that timestamp in the front to be indexed.

[2019-04-15 17:27:24.886842 -0400] rprt s=2ruc8erh8x mod=session cmd=disconnect module= rule= action= helo=ppops.net msgs=1 rcpts=1 routes=allow_relay,firewallsafe,internalnet duration=0.066 elapsed=0.622

filter {
      grok {
           match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:proofpoint}" }
       }

   kv {
     source => "proofpoint"
     field_split => " "
     include_brackets => true
     recursive => "true"
     value_split => "="
     whitespace => "strict"
   }


date {
locale => "en"
match => [ "timestamp","YYYY-MM-dd HH:mm:ss.SSS","ISO8601" ]
    }

   mutate {
     remove_field => [ "proofpoint","message","syslog_timestamp","syslog_hostname","path","@index","@version","host","port","tags" ]
   }
}

I would suggest

grok { match => { "message" => "\[(?<timestamp>[^\]]+)\] %{GREEDYDATA:proofpoint}" } }
date { locale => "en" match => [ "timestamp","YYYY-MM-dd HH:mm:ss.SSSSSS ZZ" ] }
1 Like

@Badger as always, you have come through in the clutch for me. Thank you. That allowed me to index based off of that day. I will now go ahead and focus on mapping etc. Thanks again for your help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.