Get substring of a field and use it in index name

[ldms020255]

Here's my config

input {
redis {
data_type => "list"
key => "filebeat"
host => "127.0.0.1"
port => 6379
threads => 5
type => "log"
}
}

filter {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => [ "message" , "%{NGINXACCESS}"]
}
date {
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
timezone => "Asia/Shanghai"
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "nginx-%{type}-%{+YYYY.MM.dd}"
flush_size => 1000
idle_flush_time => 2
sniffing => false
template_overwrite => true
}
}

So i assign custom date to @timestamp value. My logs are all in this Asia timezone. But the date part %{+YYYY.MM.dd} in index name is utc, which is by design i think. It actually breaks my one day log into two indices. I did search for a workaround but with no luck.

What I'm thinking is i have a "time_local" field, whose value should be something like 09/Mar/2017:00:00:03 +0800. If I can get a substring of it, 09/Mar/2017, and append to the elasticsearch index name, it would do the trick for me. Is it possible?

Thanks in advance.

[magnusbaeck (Magnus Bäck)]

Yes, use grok to capture the parts you want from time_local into a field, then reference that field in your output configuration (instead of %{+YYYY.MM.dd}). Not sure if slashes are allowed in index names, though.

It actually breaks my one day log into two indices.

Yes. So?

[ldms020255]

Thanks for the response.

So say logs Mar 08 corrupted or partially done, i want to clean up and start from fresh, normally i wound delete the index of March 08 but with the data split into different indices i can't easily make this happen.