How to transfer original @timestamp from json log to logstash

Elastic Stack Logstash
1

I am trying to setup filebeat->logstash->elasticsearch chain and I am having problems with @timestamp which is not being transferred from the logfile and a timestamp when the message arrives into logstash is used instead. I will describe it on the example below. I hope somebody will help me understanding this problem and correcting it.

Logfile has this format:

{"application":"MyTestApp","source_host":"apphost01","message":"Hello_World","@timestamp":"2017-02-14T11:38:32.257Z"}

filebeat.yml:

filebeat.prospectors:
    - input_type: log
      paths:
        - /tmp/json.log
    output.logstash:
      hosts: ["localhost:5043"]

logstash pipeline conf:

input {
        beats {
            port => "5043"
            codec => json
            type => "log4j-json"
        }
    }
    output {
         stdout { codec => rubydebug }
    }

This is the logstash's output: (notice the @timestamp is different than one in the log entry)

{
    "source_host" => "apphost01",
     "@timestamp" => 2017-02-15T14:47:57.593Z,
    "application" => "MyTestApp",
         "offset" => 118,
       "@version" => "1",
     "input_type" => "log",
           "beat" => {
        "hostname" => "tpl450",
            "name" => "tpl450",
         "version" => "5.2.1"
    },
           "host" => "tpl450",
         "source" => "/tmp/json.log",
        "message" => "Hello_World",
           "type" => "log",
           "tags" => [
        [0] "beats_input_codec_json_applied"
    ]
}
2

I think that is cause it's overwriting the original one. What if you add a date filter that ensures it's carried through?

3

I tried adding this filter, but I am not sure the filter is correctly set and it did not help:

filter {
    date {
        match => [ "@timestamp", "ISO8601" ]
        target => "@timestamp"
    }
}

The output is this:

{
    "source_host" => "apphost01",
     "@timestamp" => 2017-02-16T08:29:51.049Z,
    "application" => "MyTestApp",
         "offset" => 594,
       "@version" => "1",
     "input_type" => "log",
           "beat" => {
        "hostname" => "tpl450",
            "name" => "tpl450",
         "version" => "5.2.1"
    },
           "host" => "tpl450",
         "source" => "/tmp/json.log",
        "message" => "Hello_World",
           "type" => "log",
           "tags" => [
        [0] "beats_input_codec_json_applied",
        [1] "_dateparsefailure"
    ]
}

So the @timestamp is still not correctly set and I am even getting a _dateparsefailure.
Any help is appreciated. Thanks.

4

Any ideas how to make logstash not overwriting @timestamp json key from the original log entry?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.