Timestamp not getting my log value using date filter logstash

Elastic Stack Logstash
1

I have following config to fetch Http call and log date at time call was made, but @timestamp shows when log was send from FileBeat server to logstash one.

input {
beats {
port => 5044
}
}

filter {
if [message] !~ /Calling SMC REST API/ {
drop { }
}else {
grok {
match => ["message", "{URIPATHPARAM:request}"]
match => ["message", "%{DATA:timestamp}"]

}
date {

            match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSSS" ]
    }

}

}

output {
elasticsearch {
hosts => ["http://localhost:9200/"]
index => "smc_calls-%{+YYYY.MM.dd}"
}

}

Tried this too:

input {
beats {
port => 5044
}
}

filter {
if [message] !~ /Calling SMC REST API/ {
drop { }
}else {

    date {
            match => ["@timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
            target => "@timestamp"
    }

grok {
    match => ["message", "{URIPATHPARAM:request}"]

    }

}

}

output {
elasticsearch {
hosts => ["http://localhost:9200/"]

    }

stdout { codec => rubydebug }
}

This is message i get in console:

{
"@timestamp" => 2017-04-28T11:30:38.022Z,
"offset" => 15051398,
"@version" => "1",
"input_type" => "log",
"beat" => {
"hostname" => "host of applicaiton",
"name" => "name of host",
"version" => "5.3.1"
},
"host" => "name of host",
"source" => "log-file-name",
"message" => "[2017-04-24 06:43:12,592] @ INFO [jmsContainer-1]other message details",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_dateparsefailure",
[2] "_grokparsefailure"
]
}

Please help

2

What does the timestamp field that you're trying to parse with the date filter look like? Does the contents of that field follow the pattern of your date filter?

3

this is [2017-04-25 12:41:25,064] date format, which is start of log line which contains http url, which i have taken out.

4

You have multiple configuration problems but I don't think it's fruitful that I start listing them. I think you're trying to do too much at a time. Start with the grok filter. Get it to extract the timestamp into a field. Verify that that works. Then add the date filter and make sure its pattern matches what the timestamp field looks like. You can use the grok constructor web site to get help constructing a grok expression that matches your input.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.