# LogStash Conf | Drop Empty Lines

**URL:** <https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [December 20, 2022, 5:13pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674 "2022-12-20T17:13:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![srii](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@srii](https://discuss.elastic.co/u/srii)\
**Post date:** [December 20, 2022, 5:13pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/1 "2022-12-20T17:13:40Z")

</div>

The contents of LogStash's `conf` file looks like this:

```java
input {
    beats {
        port => 5044
    }
	
	file {
		path => "/usr/share/logstash/iway_logs/*"		
		start_position => "beginning"
		sincedb_path => "/dev/null"
		#ignore_older => 0
		codec => multiline {
         pattern => "^\[%{NOTSPACE:timestamp}\]"
         negate => true
         what => "previous"
		 max_lines => 2500
        }
  }
}

filter {	
	
	grok {
		match => { "message" => 
			['(?m)\[%{NOTSPACE:timestamp}\]%{SPACE}%{WORD:level}%{SPACE}\(%{NOTSPACE:entity}\)%{SPACE}%{GREEDYDATA:rawlog}'			
			]
		}
	}
					
	date {
	  match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSS"]
	  target => "@timestamp"
	}
		
	grok {
		match => { "entity" => ['(?:W.%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}\.%{GREEDYDATA:workerid}|W.%{GREEDYDATA:channel}\.%{GREEDYDATA:workerid}|%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}\.%{GREEDYDATA:workerid}|%{GREEDYDATA:channel}:%{GREEDYDATA:inlet}:%{GREEDYDATA:listener}|%{GREEDYDATA:channel})']
		}
	}
	
	dissect {
        mapping => {
            "[log][file][path]" => "/usr/share/logstash/iway_logs/%{serverName}#%{configName}#%{?ignore}.log"
        }
    }
	
}

output {
    elasticsearch {
        hosts => "${ELASTICSEARCH_HOST_PORT}"
		index => "iway_"
        user => "${ELASTIC_USERNAME}"
        password => "${ELASTIC_PASSWORD}"
        ssl => true
        ssl_certificate_verification => false
        cacert => "/certs/ca.crt"
    }
}

```

As one can make out, the idea is to parse a custom log employing multiline extraction. The extraction does its job. The log occasionally contains an empty first line. So:

```auto

[2022-11-29T12:23:15.073] DEBUG (manager) Generic XPath iFL functions use full XPath 1.0 syntax
[2022-11-29T12:23:15.074] DEBUG (manager) XPath 1.0 iFL functions use iWay's full syntax implementation

```

which naturally is causing Kibana to report an empty line:

[![enter image description here](https://us1.discourse-cdn.com/elastic/original/3X/8/0/80555c54141f31457c88394504c6167453f92e3c.png)](https://i.stack.imgur.com/yOTre.png)

In an attempt to supress this line from being sent to ES, I added the following as a last filter item:

```java
if ![message] {
  drop { }
}
	
if [message] =~ /^\s*$/ {
  drop { }
}

```

The resulting JSON payload to ES:

```json
{
	"@timestamp": [
		"2022-12-09T14:09:35.616Z"
	],
	"@version": [
		"1"
	],
	"@version.keyword": [
		"1"
	],
	"event.original": [
		"\r"
	],
	"event.original.keyword": [
		"\r"
	],
	"host.name": [
		"xxx"
	],
	"host.name.keyword": [
		"xxx"
	],
	"log.file.path": [
		"/usr/share/logstash/iway_logs/localhost#iCLP#iway_2022-11-29T12_23_33.log"
	],
	"log.file.path.keyword": [
		"/usr/share/logstash/iway_logs/localhost#iCLP#iway_2022-11-29T12_23_33.log"
	],
	"message": [
		"\r"
	],
	"message.keyword": [
		"\r"
	],
	"tags": [
		"_grokparsefailure"
	],
	"tags.keyword": [
		"_grokparsefailure"
	],
	"_id": "oRc494QBirnaojU7W0Uf",
	"_index": "iway_",
	"_score": null
}

```

While this does drop the empty first line, it also unfortunately interferes with the multiline operation on other lines. In other words, the multiline operation does not work anymore. What am I doing incorrectly?

---

<div class="post-metadata">

**Author:** ![RudyStolds](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@RudyStolds](https://discuss.elastic.co/u/RudyStolds)\
**Post date:** [December 20, 2022, 6:12pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/2 "2022-12-20T18:12:15Z")

</div>

I use this to drop empty messages

```auto
  # drop logs without message
  if [message] == "" { drop{} }

```

---

<div class="post-metadata">

**Author:** ![srii](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@srii](https://discuss.elastic.co/u/srii)\
**Post date:** [December 20, 2022, 6:17pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/3 "2022-12-20T18:17:17Z")

</div>

@RudyStolds Thanks but the filter to drop empty lines (I tried a similar filter) works already at the cost of messed up multiline parsing...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 20, 2022, 6:22pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/4 "2022-12-20T18:22:41Z")

</div>

> [@srii](#):
>
> ```auto
> if [message] =~ /^\s*$/ {
> drop { }
> }
> 
> ```

Your multiline expression will append blank lines to the previous line with a timestamp. `/^\s*$/` matches a blank line within a multiline message. If you just want to drop messages that only contain whitespace then try `/\A\s*\Z/`.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 20, 2022, 6:22pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/5 "2022-12-20T18:22:47Z")

</div>

> [@srii](#):
>
> Thanks but the filter to drop empty lines (I tried a similar filter) works already at the cost of messed up multiline parsing...

Can you share an example of this? This should not happen, the multiline codec happens in the `input` block of your pipeline, the `drop` filter happens in the `filter` block, after your multiline event was already created.

---

<div class="post-metadata">

**Author:** ![srii](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@srii](https://discuss.elastic.co/u/srii)\
**Post date:** [December 20, 2022, 6:37pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/6 "2022-12-20T18:37:29Z")

</div>

@Badger Thank you! That was it. The empty line is dropped.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2023, 6:37pm UTC](https://discuss.elastic.co/t/logstash-conf-drop-empty-lines/321674/7 "2023-01-17T18:37:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.