Index name from log date not from system time

Elastic Stack Logstash
1

Hi,

Today i.e on 11-Aug-2017 I am pushing back dated logs into elastic search via logstash via filebeats

I want to create index names of log dates not of current date.

Back dates logs are from Mar-Jul 2017, but for all these logs only 1 index is created by name "myindex-110817"

I want it to create separate index for each date given in log file, below is my configuration.

I want to use "mylogtime" field for creating index names

filter {
   grok {
      patterns_dir => ["config/tt_tradingserver/pattern"]	
      match => { "message" => "%{greedydata_end_nospace:mylogtime}%{zero_or_more_space}\[%{tt_ts_log_type:logtime}\]%{zero_or_more_space}\"%{zero_or_more_space}%{greedydata_end_nospace:logmsg}%{zero_or_more_space}\"" }
}

date {
   match => [ "mylogtime" , "yyyy-mm-dd hh:mm:ss.ssszz" ]
   target => "mylogtime"
   }
}

output {
   elasticsearch {
      hosts => [ "localhost:9200" ]
      index => "myindex-%{+ddmmyy}" 
  }  }
1 Like
2

The date pattern in your date filter is wrong is various ways. Please consult the documentation on which tokens to use for various timestamp components.

Can't you parse the date into the default @timestamp field (i.e. delete the target option in your date filter)? Then your current configuration will almost work out of the box (%{+ddmmyy} needs a slight adjustment).

1 Like
3

Thanks for your reply, tried as per your suggestion didnt work, if possible any small example please

4

Can you parse the date into the default @timestamp field instead of insisting on your mylogtime field?

1 Like
5

Thanks, it is working as expected, changes that I made is in GROK filter instead of "t" i used "timestamp" and I changed by date filter as

date {
	match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSSZZ" ]
}

After the above changes I was getting 2 timestamp fields

1st => @timestamp July 28th 2017, 06:09:09.062
2nd => timestamp 2017-07-28 06:09:09.062+05:30

Since i don't want to maintain 2 timestamp field, I used "remove_field" for removing timestamp.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.