Get substring of a field and use it in index name

Here's my config

input {
redis {
data_type => "list"
key => "filebeat"
host => ""
port => 6379
threads => 5
type => "log"

filter {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => [ "message" , "%{NGINXACCESS}"]
date {
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
timezone => "Asia/Shanghai"

output {
elasticsearch {
hosts => [""]
index => "nginx-%{type}-%{+YYYY.MM.dd}"
flush_size => 1000
idle_flush_time => 2
sniffing => false
template_overwrite => true

So i assign custom date to @timestamp value. My logs are all in this Asia timezone. But the date part %{+YYYY.MM.dd} in index name is utc, which is by design i think. It actually breaks my one day log into two indices. I did search for a workaround but with no luck.

What I'm thinking is i have a "time_local" field, whose value should be something like 09/Mar/2017:00:00:03 +0800. If I can get a substring of it, 09/Mar/2017, and append to the elasticsearch index name, it would do the trick for me. Is it possible?

Thanks in advance.

Yes, use grok to capture the parts you want from time_local into a field, then reference that field in your output configuration (instead of %{+YYYY.MM.dd}). Not sure if slashes are allowed in index names, though.

It actually breaks my one day log into two indices.

Yes. So?

Thanks for the response.

So say logs Mar 08 corrupted or partially done, i want to clean up and start from fresh, normally i wound delete the index of March 08 but with the data split into different indices i can't easily make this happen.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.