Log Timezone Question

Elastic Stack Logstash
1

I have been searching through the threads, but I haven't been able to fins a solution. Essentially, I have a log source that ships logs in UTC. I need to adjust them for my timezone, but changing the zone in the logstash config hasn't changed it. The logs show up in Kibana with a timestamp that is offset by the exact amount of my timezone. I'm sure it must be something simple.

Version: 7.5

2

Logs are stored in elasticsearch in UTC, so specify UTC as the timezone in the date filter.

3

I apologize, I'm new to using ELK. Do you mean use the date filter in the logstash config? Right now, I have the input section as follows:

 input {
     syslog {
         timezone => "UTC"
         port => "5514"
         type => "syslog"
         tags => [ "OS_SysLog" ]
     }
 }

If I am correct so far, do I just need to add the following?

date {
    timezone => "UTC"
}

Thanks in advance.

4

OK, I misunderstood. If your syslog entries are arriving with a timestamp in UTC then that syslog input should be OK and you would not need a date filter. They would be stored in elasticsearch as UTC and kibana (with the default settings) would translate them into the timezone where kibana runs.

5

Maybe that's the problem - my device isn't actually sending in UTC. I will verify that again. If it is actually sending in a different time zone, say EST, would the following be what is required?

date {
    timezone => "America/New_York"
}
6

Ok, making progress. The logs are in fact NOT being sent in UTC. So, should be a simple fix. I just need to build the correct date filter.

The logs come in this format:

GeneratedTime 2019/12/10 13:20:27

Edit: I previously gave the incorrect format.

7

If the date is in the right format then the syslog filter will parse it and that is where you would set the timezone. There would be no need for a date filter in that case. However in the format you have you will need to use a date filter.

8

Ok. It sends the date as follows:

GeneratedTime
2019/12/10 13:20:27

I re-read the docs on the syslog plugin and saw that logstash will only accept RFC3164, not RFC5424.

I modified my config file as shown below:

filter {
    date {
        match => ["GeneratedTime", "yyyy/MM/dd HH:mm:ss"]
        timezone => "America/New_York"	
    }
    {...}
}

I rebooted the VM to ensure all services were restarted after my config change, however, the timestamp remains unchanged when displayed in Kibana. I'm still missing something...

9

By default the date filter sets @timestamp and does not modify the parsed field, so we would expect GeneratedTime to remain a string.

If you use target to tell the date filter to overwrite GeneratedTime then the format in elasticsearch will change, however, since it has already been mapped as a string it will remain a string. But if you start over with an empty index it will be a date.

10

Ok. More progress. I have my filter config as shown:

filter {
    date {
        match => ["GeneratedTime", "yyyy/MM/dd HH:mm:ss"]
        timezone => "America/New_York"
        target => "GeneratedTime"
   }

If I set the Time Filter field name to @timestamp on the index pattern, the logs show up in the correct time. However, if I use "GeneratedTime" they do not. Is there something wrong with they way I am using the target?

Also, I have deleted the indexes after each change.

11

Hi

You need to convert the time to your timezone. This solution might help: Time in IST for log-rotation - Logstash

Hope this helps

12

Thank you for you suggestion. However, that doesn't help much in my case as I am not trying to name the files as such. I think what I need to do is overwrite my 'GeneratedTime' field with the corrected time from the same field.

13

Hi

Your case is not the same as the one in the other post, but I still think the ruby filter will do what you need.

Replace @timestamp with your GeneratedTime and filename with a variable of your choice, e.g. GeneratedTime, and you will get the time in your time zone. You'll have to play with the format in ...strftime('%Y-%m-%d/%H')), but that's it.

Hope this helps

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.