Remove blank fields from output file

[Tarun_Kalluri]

I am pretty new to using log stash, so I need to know how to remove a line which does not match the grok pattern completely from the output. I have a 500 line input out of which 4-5 lines match my pattern. Presently, it shows brackets with empty contents for all lines which do not match the pattern and valid entries are sparse in between. I am using rubydebug or json output, no luck trying drop filter or conditionals.

[lueneburger (Dirk L.)]

Hi @Tarun_Kalluri

not sure if you can drop empty lines within logstash, but you could work with IF

what kind of events? syslogs?

SYSLOGBASE is a grok pattern
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

grok {
		match => [ "message", "%{SYSLOGBASE} %{GREEDYDATA:message}" ]
		overwrite => [ "message" ]
		}

IF [program] =~ /^name$/ {
    grok {
          filter....
          }
}

at the out you doing the same with IF to only sending the logs you need to ES.

[magnusbaeck (Magnus Bäck)]

I need to know how to remove a line which does not match the grok pattern completely from the output.

filter {
  if "_grokparsefailure" in [tags] {
    drop { }
  }
}

[Tarun_Kalluri]

This is it. I had removed the "tags" field from the output, so "_grokparsefailure" did not come to me. Thanks nevertheless.