Getting _grokparsefailure when passing full log files

Elastic Stack Logstash
1
  1. have ping logs files
  2. used the excerpt of it as a sample for grok filter and tested with grok
    _debugger and all good
  3. When I pass the full file will all logs instead of excerpt get _grokparsefailure though was working when I simply used an excerpt of files.

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "/usr/share/filebeat/mylog/portal/*"
  fields:
    project_type: arcgis
- type: log
  enabled: true
  paths:
    - "/usr/share/filebeat/mylog/website/*"
  fields:
    project_type: web

#filebeat.config.modules:
#  path: /usr/share/filebeat/modules.d/*.yml

output.logstash:
  hosts: ["logstash:5044"]

processors:
  - add_docker_metadata:
      host: "unix:///host_docker/docker.sock"

Logstash.conf

input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  if "arcgis" in [fields][project_type]{
    grok {
     match => {"message" => "%{DATE_US:date} %{TImE:time} %{WORD:am_pm} \- Reply from %{IP:ip}\: bytes=32 time=%{NUMBER:timetaken:int}\ms TTL=%{NUMBER:ttl:int}"}
      }
    }

 if "web" in [fields][project_type]{
    grok {
      match => {"message" => "%{DATE_US:date} %{TIME:time} %{WORD:am_pm} \- Reply from %{IP:ip}\: bytes=32 time=%{BASE10NUM:timetaken:float}\ms TTL=%{NUMBER:ttl:int}"}
      }
    }
}

output {
  if "arcgis" in [fields][project_type]{
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "portal-test"
     }
     stdout {
            codec => rubydebug
          }
   }
  else if "web" in [fields][project_type]{
         elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "website-test"
            }
 	     stdout {
    	        codec => rubydebug
	      }
    }
  else {
 	elasticsearch {
            hosts => ["elasticsearch:9200"]
            index => "none-found"
            }
             stdout {
                codec => rubydebug
              }
   }
}

Problem:
created a file input for file beats taking excerpts from files making sure it's similar to whole files. Done this just to decrease logs size. Worked well but not when passing the whole file had an issue. Not able to figure out the exact cause.

Samples:

8/5/2021 7:31:12 AM -
8/4/2021 9:08:01 AM - Pinging 10.18.6.70 with 32 bytes of data:
8/4/2021 9:08:01 AM - Reply from 10.18.6.70: bytes=32 time<1ms TTL=127
8/4/2021 8:19:12 PM - Reply from 10.18.6.70: bytes=32 time=1ms TTL=127
8/4/2021 8:19:12 PM - 
8/4/2021 10:52:58 AM - Request timed out.
8/4/2021 8:19:12 PM - Ping statistics for 10.18.6.70:
8/4/2021 8:19:12 PM -     Packets: Sent = 3000, Received = 1000, Lost = 32 (0% loss),
8/4/2021 8:19:12 PM - Approximate round trip times in milli-seconds:
8/4/2021 8:19:12 PM -     Minimum = 0ms, Maximum = 72ms, Average = 0ms
2

Personally I would use

    dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{[@metadata][restOfLine]}" } }
    date { match => [ "[@metadata][ts]", "MM/dd/YYYY hh:mm:ss a" ] }
    if [@metadata][restOfLine] =~ /Reply from / {
        grok { match => { "message" => "Reply from %{IP:ip}\: bytes=32 time[=<]%{BASE10NUM:timetaken:float}\ms TTL=%{NUMBER:ttl:int}" } }
    }

Note the use of hh (hour of the half day) in the date pattern since I am using "a" to consume the AP/PM.

Your grok filter only matches the "Reply from" lines, so I only try to grok them. The grok uses [=<] to match both "time<1ms" and "time=1ms". Note that the grok pattern does not have to match the entire [@metadata][restOfLine] field which contains

"- Reply from 10.18.6.70: bytes=32 time<1ms TTL=127"

That config will get you packets containing

"@timestamp" => 2021-08-04T13:08:01.000Z,
 "timetaken" => 1.0,
        "ip" => "10.18.6.70",
       "ttl" => 127
3

Thanks for the better approach to parse it. But I have the same problem again with it as well.

  1. Created a new file to parse as following: copied few 100 top lines, copied tail 100 lines and copied from middle with some different format
  2. Logstash works I get the fields

when passing the full file, it's not working but I don't get the _grokparsefailure this time when using the new parser you send.

Is there a chances because the file size is large. single file is up to 5.6M

4

It is unclear what problem you are having.

5

Works for small size file but not when I pass large file.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.