Ruby date parsing

Elastic Stack Logstash
1

hi everyone,

ruby section of my config file is like below. this works fine. i set the "timeid" and i mailed to me in body section. But it sends as a UTC time value. my timezone is UTC +3. can i change it ? logstash settings file or anywhere else ?

ruby
{
code => "
event.set('timeid',event.sprintf('%{+dd MMMM YYYY HH:mm:ss.SSS}'))
"
}

.........

output {

....
email
{
......
body => "Switch says that:\nHost: %{host}\nZaman: %{timeid}\nInfo: %{message}"
}
}

output like that

Switch says that:
Host: 192.168.1.1
Zaman: 27 December 2016 13:14:21.113
Info: <187>122013: 0.0.0.0: Dec 27 16:14:21.216: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/41, changed state to down

but correct time is 27 December 2016 16:14:21.113

Sincerely

2

what is the reason for using the ruby code for date parsing and not the date filter itself?

The logstash date filter allows you to set the timezone of the local machine

For example:

% echo "12 April 2016 00:00:00.000" | bin/logstash -e "filter { date { match => ['message', 'dd MMMM YYYY HH:mm:ss.SSS'] target => timeid } }"                         
{
    "@timestamp" => 2016-12-28T18:16:44.686Z,
        "timeid" => 2016-04-11T23:00:00.000Z,
      "@version" => "1",
          "host" => "Joaos-MBP-5.lan",
       "message" => "12 April 2016 00:00:00.000",
          "type" => "stdin",
          "tags" => []
}
% echo "12 April 2016 00:00:00.000" | bin/logstash -e "filter { date { match => ['message', 'dd MMMM YYYY HH:mm:ss.SSS'] target => timeid timezone => 'Asia/Tokyo' } }"
{
    "@timestamp" => 2016-12-28T16:59:53.289Z,
        "timeid" => 2016-04-11T15:00:00.000Z,
      "@version" => "1",
          "host" => "Joaos-MBP-5.lan",
       "message" => "12 April 2016 00:00:00.000",
          "type" => "stdin",
          "tags" => []
}
3

hi João

thank you for reply.

when i used date filter , output is not human readable. and i could nout manipulate it.

output like that

first "zaman" value derived from rubycode and second one derived from timestamp value.

second "zaman" value in config file like this.

           date
                    {
                            match => ["@timestamp","ISO8601"]
                            timezone => "Europe/Istanbul"
                  }

For example for above output my server time was "Thu Dec 29 17:59:30 +03 2016"
but "@timestamp" and "timeid" value was 2016-12-28T14:59:30.012Z and 28 December 2016 14:59:30.012

it seems that date filter cannot change the timezone value.

or i misunderstanded date filter manipulation. i expect that when i change timezone by using date filter, @timestamp value should be updated that timezone.

Please correct me if i wrong.

5

As below i make some test various filter options. Please let me know what i am doing wrong

% echo "2016-12-29T10:05:30.012Z" | bin/logstash -e "filter { date { match => ['message', 'ISO8601'] target => timeid timezone => 'Etc/UTC'} }"

    {
    {
        "@timestamp" => 2016-12-29T07:22:56.555Z,
            "timeid" => 2016-12-29T10:05:30.012Z,
          "@version" => "1",
              "host" => "0.0.0.0",
           "message" => "2016-12-29T10:05:30.012Z",
              "type" => "stdin",
              "tags" => []
    }
    09:22:59.322 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}
    % echo "2016-12-29T10:05:30.012Z" | bin/logstash -e "filter { date { match => ['message', 'ISO8601']  target => timeid timezone => 'Europe/Istanbul'} }"

    {
        "@timestamp" => 2016-12-29T07:24:55.554Z,
            "timeid" => 2016-12-29T10:05:30.012Z,
          "@version" => "1",
              "host" => "0.0.0.0",
           "message" => "2016-12-29T10:05:30.012Z",
              "type" => "stdin",
              "tags" => []
    }


    % echo "2016-12-29T10:05:30.012Z" | bin/logstash -e "filter { date { match => ['@timestamp', 'ISO8601']   timezone => 'Europe/Istanbul'} }"

    {
        "@timestamp" => 2016-12-29T07:26:41.488Z,
          "@version" => "1",
              "host" => "0.0.0.0",
           "message" => "2016-12-29T10:05:30.012Z",
              "type" => "stdin",
              "tags" => [
            [0] "_dateparsefailure"
        ]
    }
    }
6

Some other tests
%echo "2016-12-29T08:02:48.695Z" | bin/logstash -e "filter { date { match => ['message', 'ISO8601'] target => 'message' timezone => 'America/New_York'} }" --log.level debug

{
    "@timestamp" => 2016-12-29T08:10:16.706Z,
      "@version" => "1",
          "host" => "0.0.0.0",
       "message" => 2016-12-29T08:02:48.695Z,
          "type" => "stdin",
          "tags" => []
}

%echo "2016-12-29T08:02:48.695Z" | bin/logstash -e "filter { date { locale => en match => ['message', 'ISO8601'] target => 'message'  timezone => 'America/New_York'} }" --log.level debug

{
    "@timestamp" => 2016-12-29T08:14:18.582Z,
      "@version" => "1",
          "host" => "0.0.0.0",
       "message" => 2016-12-29T08:02:48.695Z,
          "type" => "stdin",
          "tags" => []
}
7

or i misunderstanded date filter manipulation. i expect that when i change timezone by using date filter, @timestamp value should be updated that timezone.

No. The result of the date filter is always a UTC timestamp. The timezone option changes how the source timestamp (that is to be parsed) is interpreted.

Why do you care about the stored representation of the timestamp? Formatting the timestamp for human consumption (including timezone selection) belongs in the presentation layer, not in the database.

8

Thank you for reply Magnus.

I solved my issue by using below code. It is a not big problem . i just wanna arrange something.

ruby
{
code => "
temp=Time.new;
temp=temp.localtime.strftime ('%d %B %Y %H:%M:%S.%L %z' );
event.set('timeid',temp);
"
}

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.