Get the correct timestamp from the log file and set to @timestamp

[Loudone]

Dear All,

I'm currently trying to define log timestamp to @timestamp. But for some reasons, it seems not working properly

my input log look like:

[25/Mar/2019:08:10:56 +0100] GET "/foo/bar_" 200 1 0

On logstash side, I set the following configuration:

input {
file {
path => "/var/log/logstash/CATALINA/localhost"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "[%{{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}:mytimestamp}] %{CRON_ACTION:HTTPMETHOD} "%{URIPATHPARAM:request}" %{NUMBER:status} %{NUMBER:responseTime} %{NUMBER:bytes_size}" }
overwrite => ["message"]
}
date {
#format: 13/Jan/2016:11:55:40 +1300 (GROK: HTTP_DATE)
match => ["mytimestamp", "dd/MMM/yyyy:HH:mm:ss:Z"]
target => "@timestamp"
}
}
output {
elasticsearch {
#hosts => ["192.168.1.121:9200"]
hosts => ["172.20.10.6:9200"]
index => "access_log-%{+yyyy.MM.dd}"
}
}`

But for some reasons, the timestamp isn't correct, is not the timestamp from the log

{
"_index": "access_log-2019.03.25",
"_type": "doc",
"_id": "ip1CtmkB9Yg_cGw4eELf",
"_score": 1,
"_source": {
"tags": [
"grokparsefailure"
],
"@version": "1",
"@timestamp": "2019-03-25T19:10:02.411Z",
"host": "centos_solr_test",
"message": "[25/Mar/2019:08:10:56 +0100] GET "/v1/dck/" 200 1 0",
"path": "/var/log/logstash/CATALINA/localhost"
},
"fields": {
"@timestamp": [
"2019-03-25T19:10:02.411Z"
]
}
}

The @timestamp should be:

"@timestamp": "2019-03-25T08:10:56.100Z",

and not:

"@timestamp": "2019-03-25T19:10:02.411Z",

Do you know where is the problem?

Many thanks for your help

[Badger]

Well you are getting a _grokparsefailure tag, so you do not have a mytimestamp field. If you did you would then get a _dateparsefailure.

I would recommend using dissect rather than grok.

dissect { mapping => { "message" => '[%{mytimestamp}] %{HTTPMETHOD} "%{request}" %{status} %{responseTime} %{bytes_size}' } }

Then in the date filter use

match => ["mytimestamp", "dd/MMM/yyyy:HH:mm:ss Z"]

If you really want to use grok, then capture everything between the square brackets as one item using a custom pattern.

grok {
    match => { "message" => '\[(?<mytimestamp>[^\]]+)\] %{WORD:HTTPMETHOD} "%{URIPATHPARAM:request}" %{NUMBER:status} %{NUMBER:responseTime} %{NUMBER:bytes_size:int}' }
}

[Loudone]

Hi Badger,

your solution is working, but it's little bit strange, because I'd test before asking help my grok parsing with the kibana's grok debugger and it was working...

Anyway, I'd keep the grok parsing but I change the syntax like below:

match => { "message" => "[(?[^]]+)] %{WORD:HTTPMETHOD} "%{URIPATHPARAM:request}" %{NUMBER:status} %{NUMBER:responseTime} %{NUMBER:bytes_size}" }

Many thanks for you help

Now it's working properly