Logstash => ES, Timezone and IndexName

[bastimm]

Hello,

we have logfiles with a timestamp like "yyyyMMddHHmmss" with Europe/Berlin Timezone.
This files were sent via FileBeat=>Logstash=>ES.

Now we changed timezone in our Logstash config.
Now the timestamp in ES is correct.
But we create a new Index for each day and now there is a difference.

One short example:
timestamp: 20170401012200 => data will logged into index "log-2017.03.31", but we want the data in "log-2017.04.01"

our configs:

input {
    beats {
            port => "5044"
            host => "0.0.0.0"
    }
}

filter {
    mutate {
        add_field => {
            "shop ID" => "%{[message][0]}"
            "tpTimestamp" => "%{[message][1]}"
        }
    date {
        match => ["tpTimestamp", "yyyyMMddHHmmss"]
        timezone => "Europe/Berlin"
        target => "@timestamp"
     }
}

output {
    elasticsearch {
       hosts => ["localhost:9200"]
           index => "actionlog-%{+YYYY.MM.dd}"
       }
    }
}

Can somebody help us, how to get the right indexname with correct times?

Thanks in advance!

[sreepatch]

As per my knowledge, the indices are created on the daily basis based on your index configuration. Logstash does not bother about the time stamp in the data while filtering and sending the data to indices.
If you are using Kibana, the Date field added in the filter can be used to create a new search. Then you can filter out the data based on the time stamp in the data

[bastimm]

first thanks for your answer!

Our problem are the wrong index-names.
Search/TImestamp is like we want.
If we disable some indexes or delete some indexes, there is data from an other day included....

[magnusbaeck (Magnus Bäck)]

With

index => "actionlog-%{+YYYY.MM.dd}"

the indexes will be named after the @timestamp field which is UTC. Don't attempt to change that.

If you really need to be able to delete events on exact local time boundaries you can create another field containing the date in the local time and reference that field in the index option above.