Log date want to index as actual date for Old Log file

Hi All ,

When I am indexing data from Apache access.log , current system date ( Actual date ) is using for indexing. Actually I am indexing 2014 year logs . But these logs are indexing as in today's log . I want to index these data with actual date in log .

Using ELK 5.4.1

Indexing log from access.log

Sample data:

198.0.200.105 - - [14/Jan/2014:09:36:50 -0800] "GET /svds.com/rockandroll HTTP/1.1" 301 241 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"
198.0.200.105 - - [14/Jan/2014:09:36:50 -0800] "GET /svds.com/rockandroll/ HTTP/1.1" 200 8301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"
198.0.200.105 - - [14/Jan/2014:09:36:51 -0800] "GET /svds.com/rockandroll/js/libs/modernizr-2.6.2.min.js HTTP/1.1" 200 8768 "http://www.svds.com/rockandroll/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"

Filter is :

input{ beats
{ port => "5044"
type => "log"}
}

filter {

if [type] == "log" {
grok {
match => { "message" => "%{HOSTNAME:vhost} - - [%{HTTPDATE:timestamp}] "%{WORD:Method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} %{NUMBER:response:int} %{GREEDYDATA:Details}"}
}
mutate {
remove_field => [ "message" ]
}
}
}

output { stdout{ codec => rubydebug }
elasticsearch { hosts => ["localhost:9200"] }

The Out put format in Kibana is attached

@timestamp November 28th 2017, 16:38:56.636 ( Today time )
t timestamp 21/Oct/2014:23:19:21 -0700 ( Actual Date )

What to do for actual date comes in @timestamp

For example @timestamp = 21/Oct/2014:23:19:u7121:

HI All ,

It is solved .

Changed the filter to
if [type] == "log" {
grok {
match => { "message" => "%{HOSTNAME:vhost} - - [%{HTTPDATE:timestamp}] "%{WORD:Method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} %{NUMBER:response:int} %{GREEDYDATA:Details}"}
}
mutate {
remove_field => [ "message" ]
}
date {
locale => "en"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
timezone => "Europe/Rome"
}

}

Now indexing with actual date in 2014

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.