How to name new index based on time from log

Elastic Stack Logstash
1

Hello,

by default logstash names index using pattern logstash-%{+yyyy.MM.dd}
(and filebeat-%{+yyyy.MM.dd} for filebeat), where %{+yyyy.MM.dd} is the date when event is being published to ES.

How can I substitute date taken from my log for %{+yyyy.MM.dd} ?

For example, if I load week's old log I want an index to have a name logstash-2017.01.24 rather than today's logstash-2017.01.31?

Thanks.

2

You need to use the date filter to do that match before it gets sent to the output. That way it'll use the correct event time in that pattern.

1 Like
4

Okay, it works, thanks!

But I am facing one small problem: I have date in the following format:
2017-01-29T00:00:06

and this date is in local timezone (UTC+3). This timezone is correctly configured on server.

I have the following filter:
date {
match => [ "date", "yyyy-MM-dd'T'HH:mm:ss" ]
}

But in kibana I see that logs corresponding to 00:00:06 time are displayed as 03:00:06. Addition of "timezone" parameter to date{} filter does not change things.

Also, log lines generated after 21:00 go to test-2017.01.${DAY+1} index, which is inconvenient.

What is the proper way to configure timezone so that Kibana correctly displays time and index => "test-%{+YYYY.MM.dd}" works as expected (MM.dd corresponds to my local time)?

It looks like Logstash does not honour my local timezone (it assumes time is already in UTC), but Kinaba then converts UTC to my localtime and I get 3 hours discrepancy.

5

ES always stores in UTC.
KB adjusts that to whatever TZ the browser is set as.
LS also assumes UTC and then uses that when talking to ES, so the "rollover" time is 0000UTC.

6

Okay, this explains things. But what is the solution? How can I tell LS the actual TZ of my timestamp, so it correctly converts it to UTC?

Index names are not so important, but at least store correct time in my "date" field so that graphs at Kibana do not have 3 hours shift ahead of actual time.

7

Okay, this explains things. But what is the solution? How can I tell LS the actual TZ of my timestamp, so it correctly converts it to UTC?

If the system's timezone isn't detected automatically use the date filter's timezone option.

8

As I already wrote in this thread, timezone is correctly configured on my computer and I tried to set timezone parameter directly. No success.

I used timezone value in formats "Asia/Qatar" and "Etc/GMT-3" according to
http://joda-time.sourceforge.net/timezones.html

Consider this log line:
2017-01-29T00:00:06 189 200 127.0.0.1 GET /

and

date {
match => [ "date", "yyyy-MM-dd'T'HH:mm:ss" ]
timezone => "Etc/GMT-3"
}

Logstash names index as test-2017.01.28 (28 because Jan 29 0:00 +003 is Jan 28 21:00 UTC).

But Kibana displays this line at 03:00 time.

In Time column: January 29th 2017, 03:00:06.000
In _source column: date:January 29th 2017, 03:00:06.000 offset:...
Below in Table tab:
@timestamp January 29th 2017, 00:00:06.000
date January 29th 2017, 03:00:06.000

But in JSON tab:
"_source": {
"date": "2017-01-29T00:00:06",

So looks like despite timezone => "Etc/GMT-3" ES still treats this time as 00:00 UTC rather that 00:00 Etc/GMT-3

9

Unless you set target => "date" Logstash will store the parsed timestamp in the @timestamp field (which is the one used when expanding %{+yyyy.MM.dd}). AFAICT Logstash parses your timestamp and sets @timestamp correctly:

[magnusbk@lnxolofon] /tmp/trash.vxiU$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  date {
    match => [ "message", "yyyy-MM-dd'T'HH:mm:ss" ]
    timezone => "Etc/GMT-3"
  }
}
$ echo '2017-01-29T00:00:06' | /opt/logstash/bin/logstash -f test.config 
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "2017-01-29T00:00:06",
      "@version" => "1",
    "@timestamp" => "2017-01-28T21:00:06.000Z",
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
10

Aha, I see now.

How can I instruct logstash to update both @timestamp and "date" to adjust timezone? Can I use array as target value?
target => ["date", "@timestamp"]

1 Like
11

No, you need two date filters.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.