Monthly indexes

[goncalobsantos]

I am using Logstash to create monthly indexes using the following format in the .conf file:

output{
elasticsearch{
index => "index_name-%{+YYYY.MM}"
...
}
}

This format places most events in the correct month. Yet it places some events in incorrect months. For instance, I have an index_name-2020.07 with 100000 events. But I also have an index_name-2020.06 with 20 events that should be placed in index_name-2020.07 and an index_name-2020.08 with 100 events that should be placed in index_name-2020.07.

I haven't been able to find a common pattern that explains this behaviour. I initially assumed that this had to be related with my current timezone. All events in index_name-2020.06 all occur between 01/07/2020 0:00 and 1:00. However, there are events in index_name-2020.7 that also occur within that time period. Moreover, there are events in index_name-2020.8 that occur all throughout 2020.07 (even if most of those occur in 31/07/2020)

Maybe this has to do with my current timezone, but I haven't found a way to test this hipothesis.

Thanks in advance

[Badger]

@goncalobsantos

index => "index_name-%{+YYYY.MM}"

A date reference in an index name refers to [@timestamp], and [@timestamp] is in UTC, so if your local timezone is not UTC you should expect some events at the end or beginning of the month to appear in the "wrong" index.

[goncalobsantos]

Thank you for your reply Badger. I now understand what's happening
I assume that the answer will be negative, but do you know if it's possible to edit the output block (or some other part of the .conf file) to make it sensitive to this sort of timezone, resulting in monthly indices that agree with it? Or, even better, if I can make the date reference in an index name, refer to a different timefield?

[Badger]

The date in a sprintf reference always uses @timestamp, you cannot tell it to use another field.

elasticsearch is designed to work with UTC timestamps. Kibana maps these to the browser's timezone by default.

You can have logstash fib about what timezone the @timestamp field is in, but that will break things. Slight errors in timestamp around the start of daylight savings for example, but as I recall there were other (minor) things that simply did not work when I tried this. That was a couple of years back so I am not sure what the current state is.