Logstahs behavior about empty lines

Elastic Stack Logstash
1

Hello,

I want know how logstash interprete empty lines in a flat file. I must to place a particular option (see my filter below) or by itself he will ignore them?

I retrive file and I apply a csv filter on it. I delete empty line with if [message] =~ /^\s$/* , and I delete the header (if it finds a word of it) with "\brows\b"

Look my filter :

filter {
  if [type] == "lertg" {
    csv {
      separator => "|"
      skip_empty_columns => true
      columns => [ "........" ]
    }
    if [message] =~ "\bPOID\b" {
      drop { }
    }
    if [message] =~ "\brows\b" {
      drop { }
    }
    if [message] =~ /^\s*$/ {
      drop { }
    }
    date {
      match => [ "CREATED" , "UNIX" ]
      remove_field => ["CREATED_T"]
    }
  }
}

Wht do you think ?

Sometimes If no value is entered in one columns, The skip empty columns is good option or it' useless ?

2

If you want Logstash to ignore empty lines you need to drop them explicitly. Logstash doesn't do anything on its own here.

3

Ok @magnusbaeck , so

if [message] =~ /^\s*$/ {
  drop { }

is the good method ?

4

Yes, that's fine.

5 
if [message] =~ "/^\s*$/ {"
  drop { }
}

@magnusbaeck ?

Because it's return me this error message aha

SyntaxError: (eval):409: syntax error, unexpected null
Aug 31 15:47:37 opm1zlog01 logstash: if (((event.get("[message]") =~ //^\s*$//))) # if [message] =~ "/^\s*$/"
Aug 31 15:47:37 opm1zlog01 logstash: ^
Aug 31 15:47:37 opm1zlog01 logstash: eval at org/jruby/RubyKernel.java:1079

Besides, what is this syntax "if (((event.get("[message]") =~ //^\s*$//)))" ? Is she better than the other?

Edit, with new syntaxe, i have a logs ! :slight_smile:

Cannot create pipeline {:reason=>"Expected one of #, ( at line 96, column 16 (byte 2722) after filter {\n if [type] == \"lertg\" {\n csv {\n separator => \"|\"\n columns => [ \"POID\",\"CREATED\",\"MSN\",\"PERCENT\",\"MESSAGE\",\"TEXT\",\"SOME\",\"STATUS\",\"TYPE\" ]\n }\n if [message] =~ \"\\bPOID\\b\" {\n drop { }\n }\n if [message] =~ \"\\brows\\b\" {\n drop { }\n }\n if [message] =~ \"\\bpagesize\\b\" {\n drop { }\n }\n if (((event"}

My conf file have only 42 lines ..... 96 is a bullshit

6

if [message] =~ "/^\s*$/ {"

No, that's not correct.

Besides, what is this syntax “if (((event.get(”[message]") =~ //^\s*$//)))" ? Is she better than the other?

Ignore that. It's a bad error message.

My conf file have only 42 lines … 96 is a bullshit

Do you have more than one file in /etc/logstash/conf.d (or wherever you store your config files)?

7

Ok, i see.

I handed the basic formula but it spits too many errors with in the logs :

If i put that :

if [message] =~ "/^\s*$/" {
      drop { }
    }

It's nok, Logstash don't start

If i put that :

if [message] =~ /^\s*$/ {
      drop { }
    }

Logstash begin to work after many many many many error look like :

[2017-08-31T16:50:35,434][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,434][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,459][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,459][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,460][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,475][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,476][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,476][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,484][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,485][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,485][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,495][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,496][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,497][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,506][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,507][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,509][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,518][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,519][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,520][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,541][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,541][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,551][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,553][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,562][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,564][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,573][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,575][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,600][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,620][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,635][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,649][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,663][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,676][WARN ][logstash.filters.csv     ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}

In my log, i have many many many blank line.... i don't know how resolve it.

8

Filters are processed in order so your csv filter must come after the drop filter.

9

Oh ! Thank you i have never think about that

Thanks a lot(s)

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.