Logstash: error when use date filter

Elastic Stack Logstash
1

Hi, I'm setting up an ELK Stack to process some logs that are sent to us from AKAMAI. An example line:

2020-11-18 14:58:27 2.17.200.11 - - - - GET /unaurl.es/N3X3QDOPYNHGPO5R6ZYCEOWCNM.png - 200 1 36513 848 1 80 HTTP/1.1 "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AkamaiImageServer VelocitudeMP/1.0;IM/1.0" "-" "-"

Config file:

input {
    file {
        path => "/var/data/logs/*"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
    grok {
        patterns_dir => ["/etc/logstash/conf.d/patterns"]
        match => {
            "message" => "%{TIMESTAMP_ISO8601:timestamp} %{IPV4:c-ip} %{USERNAME:cs-username} %{DATA:s-sitename} %{DATA:s-computername} %{CUSTOMIP:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{CUSTOMURIPATH:cs-uri-query} %{NUMBER:sc-status} %{NUMBER:sc-win32-status} %{NUMBER:sc-bytes} %{NUMBER:cs-bytes} %{NUMBER:time-taken} %{CUSTOMPORT:s-port} %{DATA:cs-protocol} %{QS:cs-user-agent} %{DATA:cs-cookie} %{DATA:cs-referer}"
        }
    }
    date {
        match => ["timestamp", "YYYY-MM-dd HH:mm:ss"]
        target => "@timestamp"
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "index-%{+YYYY.MM.dd}"
    }
    stdout {
        codec => rubydebug
    }
}

Patterns file:

CUSTOMIP (?:%{IP}|-)
CUSTOMURIPATH (?:%{URIPATH}|-)
CUSTOMPORT (?:%{POSINT}|-)

The problem is that if I remove the date filter, all the fields are processed but of course the timestamp that catches me is the moment of reading the file.

If I leave the date filter, it gives me a grok parse failure, which I do not understand because it parses well when date filter is gone.

2

Sorry, the problem is that some lines in logs not matches grok filter. The filter date is correct.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.