Replacing @timestamp with a custom field does not occur

Elastic Stack Logstash
1

Hello,

I am currently using the HTTP-IN plugin to receive HTTP PUT requests from various systems.

The systems have a field called CreationTime which I am trying to use instead of the @timestamp field.

I have read few threads and none of the fixes seemed to help and the @timestamp is not being replaced.

The CreationTime test value is:

21/10/2016 15:13:06

My configuration file contents:

input {
  http{
   port => 8080
  }
}
filter {
   date {
        match => ["CreationTime", "dd/MM/YYYY HH:mm:ss"]
        target => "@timestamp"
   }
}
output {
  elasticsearch {
    hosts => "127.0.0.1"
    codec => "json"
    index => "logstash-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

If someone could give me some pointers then I would very much appreciate that.

2

Your date filter looks correct if your events indeed have a CreationTime field containing a date in that format. Please give an example of an event, e.g. one captured by your stdout { codec => rubydebug } output.

3

@magnusbaeck thanks for your reply, the below is the output I am getting in logstash, as you can see the @timestamp is not replaced by CreationTime but you may be able to point out if something is incorrect:

> JobId:752fc480
> StartedBy:live.com#user@outlook.com
> Exception: - 
> StatusDetails:None  
> Errors: - 
> Output: - 
> Status:Completed 
> Warnings: - 
> EndTime:21/10/2016-15:16:36 
> CreationTime:21/10/2016 15:13:06  
> runOn: - 
> Verbose: - 
> @version:1 
> @timestamp:October 21st 2016, 15:23:30.275 
> host:<IP>
> headers.request_method:PUT 
> headers.request_path:/azure/jobs/ 
> headers.request_uri:/azure/jobs/ 
> headers.http_version:HTTP/1.1 
> headers.http_user_agent:Mozilla/5.0 (Windows NT; Windows NT 6.1; en-GB) WindowsPowerShell/5.0.10586.117 
> headers.content_type:application/json 
> headers.http_host:<somehost>.cloudapp.net:8080` 
> headers.content_length:673 
> _id:AVfnoBak2aiUzdk-hbNJ 
> _type:logs 
> _index:logstash-2016.10.21
4

Is that the payload you are receiving? What is the output from the Logstash stdout output using the ruby debug codec?

5

This is exactly what I get in Kibana, which I am assuming is stored in Elastic search via logstash,

6

If that is the raw message, you have not parsed out any fields, especially not the CreationTime that the date filter expects. You probably need to apply a grok filter or possible a kv filter to parse out the fields in Logstash before you try to use the date filter.

7

Thank you for your help. I am quite a novice in all honesty. I am not sure on what the grok filter is and what is its function and how to implement it.