Convert the default ES UTC time to local timezone with logstash

Elastic Stack Logstash
1

Hello everyone,

My goal is export alerts from elasticsearch and I have 2 problems left to solve:
My current configuration:

input {
  elasticsearch {
    hosts => "https://ip:9200"
    index => ".internal.alerts-security.alerts-default-*"
    user => "user"
    password => "pass"
    ssl_certificate_verification => false
    query => '{ "query": { "match": { "event.dataset": "endpoint.alerts" } }, "sort": [ "_doc" ] }'
    schedule => "* * * * *"
    }
}

output {
  file {
    path => "/tmp/output.txt"
  }
}

1.How can I convert the default @timestamp (UTC) field coming from elastic to my timezone?
2.How can I query documents only in the last x minutes?(With the current configuration logstash retrieves every document every time it runs.)

2

There is an example of that here.

3

Can you exaplain what it does?
For my understanding it just creates an rt field and sets the localtime with a value +8 hours from that date.

4

That's right. If a document has an [rt] field in UTC it creates another field in localtime (provided localtime is MYT -- it would have to be adjusted for other local timezones).

When you asked "How can I convert the default @timestamp (UTC) field coming from elastic to my timezone?" I thought that was what you wanted.

5

My goal is to have a field that contains the timestamp coming from the @timestamp field(utc) in my local timezone gmt+2.
I made some changes to the configuration,it creates the timestamp_zoned field but the date is the same as in the original @timestamp field. :

filter {


    mutate {
        add_field => {
            "timestamp_zoned" => "%{@timestamp}"
        }
    }

    date {
        match => [ "timestamp_zoned", "yyyy-MM-dd HH:mm:ss'Z'" ]
        timezone => "Europe/Budapest"
        target => "timestamp_zoned"
    }


}
6

The Z at the end of the [timestamp_zoned] indicates it is in Zulu (UTC). The timezone option on the date filter is ignored in that case.

7

Thanks for the information I didnt know that.
I removed it but nothing changed :frowning: .

8

Any idea?
I have none, I tried everything I could find to solve this problem.

9

What exactly is the problem? You described an issue and I provided a link to what I thought was a solution. If it is not a solution then I need a better problem definition.

10

I receive a UTC timestamp from elastic in the @timestamp field.
My goal is to create a new field from that timestamp field in my timezone(GMT+2).
For example:
I have the following the document that is coming from ES.

    "@timestamp": [
      "2024-06-11T14:58:20.420Z"
    ],

I want to have a field that contains the time in my local timezone(+2 hours difference):

    "localtime": [
      "2024-06-11T16:58:20.420Z"
    ],
11

OK, so you need to tweak the ruby code slightly

input { generator { count => 1 lines => [ '{ "@timestamp": "2024-06-11T14:58:20.420Z" }' ] codec => json } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    ruby {
        code => '
            t = event.get("[@timestamp]").to_f
            t = Time.at(t, in: "+02:00")
            event.set("localtime", t.strftime("%Y-%m-%dT%H:%M:%S.%LZ"))
        '
    }
    date { match => [ "localtime", "ISO8601" ] target => "localtime_as_timestamp" }
}

will produce

             "localtime" => "2024-06-11T16:58:20.420Z",
            "@timestamp" => 2024-06-11T14:58:20.420Z,
"localtime_as_timestamp" => 2024-06-11T16:58:20.420Z
12

It works!

Thanks a lot!