Date Filter Logstash

Elastic Stack Logstash
1

Hi everyone!

I am having a little issue trying to get right the date filter in logstash.
What I am trying to archive here is to change the @timestamp for the actual log time.

My logs looks like this...

2020-02-06 00:11:09 192.168.X.X GET /dispatcher/DSP_posicione.........................

My filter bit is:

filter {
if "Access" in [tags] {
	grok {
		match=> [ "message", "%{log_timestamp:fechalog} %{GREEDYDATA:mensaje}" ]
	}
    date {
    	match => ["fechalog", "yyyy-MM-dd HH:mm:ss" ]
        target => "@timestamp"
	} 
	dissect {
		mapping => {
			message => '%{log_timestamp} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} - %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}'
			}
	} 
	mutate {
		add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
	}
	mutate {
		remove_tag => ["beats_input_codec_plain_applied"] 
    	}
} else {
	grok {
		match=> [ "message", "%{log_timestamp:fechalog} %{GREEDYDATA:mensaje}" ]
	}
    date {
    	match => ["fechalog", "yyyy-MM-dd HH:mm:ss" ]
        target => "@timestamp"
	} 
	dissect {
		mapping => {
			message => '%{log_timestamp} %{c-ip} %{c-port} %{s-ip} %{s-port} %{cs-version} %{cs-method} %{cs-uri} %{sc-status} %{s-siteid} %{s-reason} %{s-queuename}'
		}
	}
	date {
    		match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss" ]
	        target => "@timestamp"
	} 
	mutate {
		add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
	}
	mutate {
		remove_tag => ["beats_input_codec_plain_applied"] 
    	}
} }

Logstash don't give me an error but neither does it.

Here is a bit of the logstash output, he knows what is log_timestamp but it does not make the change.

 "@timestamp" => 2020-05-11T13:43:47.232Z,
            "agent" => {
             "version" => "7.6.2",
                  "id" => "09793bd2-b7ec-476c-80d8-df3b15c1a93b",
                "type" => "filebeat",
        "ephemeral_id" => "b902e8f6-725d-43b0-acf4-c2b8779cf188",
            "hostname" => "webapps01"
    },
    "log_timestamp" => "2018-05-12 05:43:55"

Do not know where is my mistake here, I do appreciatte any help you could give me.

Thank you in advance.

2

The mismatch between log_timestamp and @timestamp would be expected if "Access" in [tags], since in that case you do not call the date filter.

3

Hi Badger, thank you very much for your reply. Sorry but i do not get what you are telling me. What you mean with that a i do not call the date filter. I do call the date filter right under the grok filter. Sorry for my simplicity....

4

Hi Badger,

I did start again as I am sure it was a little mistake that i couldn't see.

So I break apart my pipeline and I did manage to change the @timestamp for the actual time of the log but it's not completely the same.

   "fechalog" => "2018-06-30 00:35:16",
"@timestamp" => 2018-06-29T22:35:16.000Z,
     "agent" => {
         "version" => "7.6.2",
            "type" => "filebeat",
              "id" => "09793bd2-b7ec-476c-80d8-df3b15c1a93b",
        "hostname" => "webapps01",
    "ephemeral_id" => "8fe3af77-8247-42f6-855b-5dce623a6ce5"

So as you can see the fechalog and the @timestamp are almost the same but don't know why it changes the day and the hour, the other bit is fine. Why is that?

Here is my new pipeline....

filter {
	if "Access" in [tags] {
		grok {
			match=> [ "message", "%{TIMESTAMP_ISO8601:fechalog} %{GREEDYDATA:mensaje}" ]
		}
		date {
			match => ["fechalog", "yyyy-MM-dd HH:mm:ss" ]
		}
} else {
		grok {
			match=> [ "message", "%{TIMESTAMP_ISO8601:fechalog} %{GREEDYDATA:mensaje}" ]
		}
		date {
			match => ["fechalog", "yyyy-MM-dd HH:mm:ss" ]
		}
}

Why the date filter change my day and the hour and does not respect it??

Thanks

5

Done, it was as simple as add timezone => "UTC" inside the date filter. That do the trick!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.