Logstash cannot assign correct date to log timestamp

Elastic Stack Logstash
1

Could anybody clarify if it's possible to add date part to log timestamp in Logstash not in 'UTC' but in servers time zone.

For example, if I have log record timestamp like this (without date part):

21:17:43,124 INFO [...

and I want to add todays date to every log record timestamp and assign that value to @timestamp field.

filter {
    grok {
      match => {"message" => "(?<logTime>[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}) "}
    }
    mutate {
      add_field => {"logTimestamp" => "%{+YYYY-MM-dd} %{logTime}"}
    }
    date {
      match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
      timezone => "..."
    }
}

However, it will add date in 'UTC' time zone and if host machine is in different time zone, @timestamp field will show tomorrow date instead of today.

Can it be Logstash bug?

2

Could anybody clarify if it's possible to add date part to log timestamp in Logstash not in 'UTC' but in servers time zone.

No, that's not possible. @timestamp is always UTC.

and I want to add todays date to every log record timestamp and assign that value to @timestamp field.

That assumes that you always process the file in real time on the exact same day that the log record was produced. If that really is an okay assumption to make you can use a ruby filter to produce today's date in your local timezone.

3

@magnusbaeck Thanks.

Is this correct config:

filter {
    grok {
      match => {"message" => "(?<logTime>[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}) "}
    }
    ruby {
      code => "event['logTimestamp'] = Time.now.strftime('%Y-%m-%d') + ' ' + event['logTime']"
    }
    date {
      match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
      timezone => "..."
    }
}

I replaced mutate filter with ruby one to get the current date of the log.

4

That looks okay for Logstash 1.x and 2.x but for 5.x and later you need to use the new event API (https://www.elastic.co/guide/en/logstash/current/event-api.html).

5

I think ruby filter should look like this:

ruby {
  code => 'event.set("logTimestamp", Time.now.strftime('%Y-%m-%d') + ' ' + event.get("logTime"))'
}

But Logstash complains about Expected one of #, {, } for date pattern. Any hints what can be wrong?

6

Can't tell without seeing the whole configuration, but you can try commenting out pieces of it to narrow the problem down.

1 Like
7

This is complete filter section:

filter {
  grok {
    match => {"message" => "(?<logTime>[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})"}
  }
  ruby {
    code => 'event.set("logTimestamp", Time.now.strftime('%Y-%m-%d') + ' ' + event.get("logTime"))'
  }
  date {
    match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
    timezone => "America/Chicago"
  }
}

And Logstash complains about this part('%Y-%m-%d')

I cannot find any examples in docs, not sure what could be wrong with syntax.

8

I changed it to:

filter {
    grok {
      match => {"message" => "(?<logTime>[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})"}
    }
    ruby {
      code => 'event.set("currentDate", Time.now.strftime("%Y.%m.%d"))'
    }
    ruby {
      code => 'event.set("logTimestamp", event.get("currentDate") + " " + event.get("logTime"))'
    }
    date {
      match => ["logTimestamp", "YYYY.MM.dd HH:mm:ss,SSS"]
      timezone => "America/Chicago"
    }
}

but it still shows date in 'UTC' time zone.

I also tried this to get current date:

ruby {
  code => 'event.set("currentTime", Time.now.getlocal("-07:00"))'
}

but the date is in 'UTC' time zone.

What' is the correct way to get current local date?

9

And Logstash complains about this part ('%Y-%m-%d')

Aha. Don't use single quotes within the single-quoted Ruby code snippet. Use double quotes everywhere within the string.

but it still shows date in 'UTC' time zone.

As I said @timestamp is always UTC.

10

As I said @timestamp is always UTC.

Does that mean there is no way to change @timestamp to another time zone?

I would like to see my logs time in my current timezone, not in UTC
If the log was created today at 21:00:00,000 I don't want to add tomorrow date to its timestamp, It should have today date in timestamp.

11

Does that mean there is no way to change @timestamp to another time zone?

On the ES side it'll always be stored as UTC and that's what the date filter produces (not configurable). Reading the ES documentation for the date type it seems it accepts timestamps with a non-zero UTC offset, i.e. your documents can contain timestamps in local time as long as the offset is specified. I haven't tried it because I think UTC is a good thing that people should stop fighting.

I would like to see my logs time in my current timezone, not in UTC

See where? Kibana converts the timestamps to the browser's timezone.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.