# Logstahs behavior about empty lines

**URL:** <https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971>\
**Category:** Logstash\
**Created:** [August 31, 2017, 8:38am UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971 "2017-08-31T08:38:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [August 31, 2017, 8:38am UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/1 "2017-08-31T08:38:30Z")

</div>

Hello,

I want know how logstash interprete empty lines in a flat file. I must to place a particular option (see my filter below) or by itself he will ignore them?

I retrive file and I apply a csv filter on it. I delete empty line with _if [message] =~ /^\s$/_\* , and I delete the header (if it finds a word of it) with **"\brows\b"**

Look my filter :

```
filter {
  if [type] == "lertg" {
    csv {
      separator => "|"
      skip_empty_columns => true
      columns => ["........"]
    }
    if [message] =~ "\bPOID\b" {
      drop { }
    }
    if [message] =~ "\brows\b" {
      drop { }
    }
    if [message] =~ /^\s*$/ {
      drop { }
    }
    date {
      match => ["CREATED" , "UNIX"]
      remove_field => ["CREATED_T"]
    }
  }
}

```

Wht do you think ?

Sometimes If no value is entered in one columns, **The skip empty columns** is good option or it' useless ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 31, 2017, 8:55am UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/2 "2017-08-31T08:55:39Z")

</div>

If you want Logstash to ignore empty lines you need to drop them explicitly. Logstash doesn't do anything on its own here.

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [August 31, 2017, 9:26am UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/3 "2017-08-31T09:26:51Z")

</div>

> [@magnusbaeck](#):
>
> If you want Logstash to ignore empty lines you need to drop them explicitly. Logstash doesn’t do anything on its own here.

Ok @magnusbaeck , so

```
if [message] =~ /^\s*$/ {
  drop { }

```

is the good method ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 31, 2017, 9:44am UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/4 "2017-08-31T09:44:50Z")

</div>

Yes, that's fine.

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [August 31, 2017, 1:54pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/5 "2017-08-31T13:54:24Z")

</div>

```
if [message] =~ "/^\s*$/ {"
  drop { }
}

```

@magnusbaeck ?

Because it's return me this error message aha

```
SyntaxError: (eval):409: syntax error, unexpected null
Aug 31 15:47:37 opm1zlog01 logstash: if (((event.get("[message]") =~ //^\s*$//))) # if [message] =~ "/^\s*$/"
Aug 31 15:47:37 opm1zlog01 logstash: ^
Aug 31 15:47:37 opm1zlog01 logstash: eval at org/jruby/RubyKernel.java:1079

```

Besides, what is this syntax "if (((event.get("[message]") =~ //^\s\*$//)))" ? Is she better than the other?

Edit, with new syntaxe, i have a logs ! 🙂

`Cannot create pipeline {:reason=>"Expected one of #, ( at line 96, column 16 (byte 2722) after filter {\n if [type] == \"lertg\" {\n csv {\n separator => \"|\"\n columns => [\"POID\",\"CREATED\",\"MSN\",\"PERCENT\",\"MESSAGE\",\"TEXT\",\"SOME\",\"STATUS\",\"TYPE\"]\n }\n if [message] =~ \"\\bPOID\\b\" {\n drop { }\n }\n if [message] =~ \"\\brows\\b\" {\n drop { }\n }\n if [message] =~ \"\\bpagesize\\b\" {\n drop { }\n }\n if (((event"}`

My conf file have only 42 lines ..... 96 is a bullshit

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 31, 2017, 2:41pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/6 "2017-08-31T14:41:54Z")

</div>

> if [message] =~ "/^\s\*$/ {"

No, that's not correct.

> Besides, what is this syntax “if (((event.get(”[message]") =~ //^\s\*$//)))" ? Is she better than the other?

Ignore that. It's a bad error message.

> My conf file have only 42 lines … 96 is a bullshit

Do you have more than one file in /etc/logstash/conf.d (or wherever you store your config files)?

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [August 31, 2017, 2:53pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/7 "2017-08-31T14:53:35Z")

</div>

Ok, i see.

I handed the basic formula but it spits too many errors with in the logs :

If i put that :

```
if [message] =~ "/^\s*$/" {
      drop { }
    }

```

It's nok, Logstash don't start

If i put that :

```
if [message] =~ /^\s*$/ {
      drop { }
    }

```

Logstash begin to work after many many many many error look like :

```
[2017-08-31T16:50:35,434][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,434][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,459][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,459][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,460][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,475][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,476][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,476][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,484][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,485][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,485][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,495][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,496][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,497][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,506][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,507][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,509][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,518][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,519][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,520][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,541][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,541][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,551][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,553][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,562][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,564][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,573][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,575][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,600][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,620][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,635][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,649][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,663][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[2017-08-31T16:50:35,676][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}

```

In my log, i have many many many blank line.... i don't know how resolve it.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 31, 2017, 3:05pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/8 "2017-08-31T15:05:55Z")

</div>

Filters are processed in order so your csv filter must come _after_ the drop filter.

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [August 31, 2017, 3:11pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/9 "2017-08-31T15:11:22Z")

</div>

Oh ! Thank you i have never think about that

Thanks a lot(s)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 28, 2017, 3:11pm UTC](https://discuss.elastic.co/t/logstahs-behavior-about-empty-lines/98971/10 "2017-09-28T15:11:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.