Create daily indices based on local timezone

Elastic Stack Logstash
1

While trying to figure out how to make indices rollover at midnight local time, I found a bunch of people asking how to do it but no clear solution.

The ruby code below generates a YYYY.MM.dd formatted string based on @timestamp in the specified timezone (in my case Australia/Melbourne). The offset is calculated in a separate step to be transparent with daylight savings.

  ruby {
    code => "
      require 'tzinfo'
      tz = 'Australia/Melbourne'
      offset = TZInfo::Timezone.get(tz).current_period.utc_total_offset / (60*60)
      event.set('[@metadata][index_day]', event.get('@timestamp').time.localtime('+' + offset.to_s + ':00').strftime('%Y.%m.%d'))
      "
  }

Then -%{[@metadata][index_day]} can be used in elasticsearch output instead of %{+YYYY.MM.dd}.

I had a bit of a look at the rollover API, and I suspect xpack's ILM might take care of this use case, but I wanted something simple that Just Works with Logstash OSS.

Hope it helps someone!

2

You can use date filter with logstash to grand any event date from source and map it to @timestamp, if no source field with the date logstash will populate the @timestamp with the system date

## Convert event date time into @timestamp
date { 
		match => ["TimeStamp", "yyyyMMddHHmmss"]
		timezone => "UTC"
		target => "@timestamp"
		}

Then in the output, append any date format to your index name

   elasticsearch {
    hosts => "http://localhost:9200"
    index => "index-name-%{+YYYY-MM-dd}"
}
3

I think you've missed the point, which is: to ensure indices rollover at midnight localtime in a non-UTC timezone, including during summer time.

4

If your servers clock is set to local time what is wrong with

    event.set("[@metadata][indexName]", Time.now.strftime("%Y.%m.%d"))

local time -- check. daylight savings handled -- check.

1 Like
5

@Badger that would be fine under those circumstances, but the server's time is UTC. Which I guess would be a pretty common scenario.

Yours in definitely a good point to have in this thread for posterity though!

6

You could parse it once telling the date filter it is UTC, then parse again as local time, but that loses the DST handling

    date { match => [ "message", "YYYY-MM-dd HH:mm" ] }
    mutate { add_field => { "[@metadata][indexName]" => "%{+YYYY.MM.dd}" } }
    date { match => [ "message", "YYYY-MM-dd HH:mm" ] timezone => "Etc/GMT+4" }

I think you have to parse it as the local timezone to get the right DST handling, so the question you are really asking is how to get ruby (not ruby on rails) to display a timestamp in a non-local timezone. I do not know the answer to that.

7

My post was a solution to this problem, rather than a question. Is there a problem with it I'm not seeing? (apart from not handling sub-hour TZ offsets in its present form)

8

No, I misunderstood the question. Thanks for posting a solution.

9

The ruby code initially used utc_offset (which doesn't take into account DST) rather than utc_total_offset (which does).

I've fixed the example in the OP.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.