Date filter does not work

Elastic Stack Logstash
1

I use grok and date filter to set correct @timestamp. It works in simple case. But when I added if condition @timestamp no longer installed from the message.

filter {
  if 'backend.info.log' in [log][file][path] {
    grok {
      match => [
        "message", '(?m)%{LOGLEVEL:log-level} %{TIMESTAMP_ISO8601:timestamp}.*Пользователь - (?<user>[^;]+).*id - (?<user-id>[^;]+).*email - (?<email>[^;]+).*Агент клиента -(?<useragent>[^;]+).*ip - %{IP:client-ip}.*token: (?<token>[^\n|^\s]+).*Запрос: \"(?<request>[^\s^\"]+).*Метод: \"(?<method>[\w]+).*Параметры: (?<params>\{.*\}).*Модуль: (?<module>[^;]+).*Функция: (?<func>[^;]+), # backend.info.log
      #### Other regexe #####
      ]
    }
  } else {
    grok {
      match => [
        "message", "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:log-level}\]%{GREEDYDATA:message}", # rabbitmq.info

        "message", "%{LOGLEVEL:log-level} %{TIMESTAMP_ISO8601:timestamp}", # gunicorn.error

            #### Other regexe #####
      ]
    }
  }
  date {
      match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
      timezone => "Europe/Moscow"
    }
}

I know about multiple regex inside the match, but I have a legacy logs in various formats and and I want to optimize logstash work time by using if condition.

2

I think where you have message it should be timestamp since that is the time field you have extracted out of the message

date {
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
      timezone => "Europe/Moscow"
    }
1 Like
3

Thanks. You are right. Too much time on debugging blunted my attention :frowning:

4

:slight_smile: Totally....

We stare at the same screen... same code... Always good to get another set of eyes or take a break... and come back!

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.