# Logstash vs rsyslog java stack multiline issue

**URL:** <https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557>\
**Category:** Logstash\
**Created:** [August 18, 2017, 9:59am UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557 "2017-08-18T09:59:23Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 9:59am UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/1 "2017-08-18T09:59:23Z")

</div>

Hi All,  
Does someone configure a multiline event starting from rsyslog messages? We tried to implement a catch all logstash instance for all the messages forwarded by docker daemons via syslog but we can't parse the multiline events.

The our configuration is 🙂

```
input {
  syslog {
    type => "docker-dev-support"
    host => "0.0.0.0"
    port => 6000
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => false
      what => "previous"
    }
  }
} 

```

Thanks,  
Marcello

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 18, 2017, 11:10am UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/2 "2017-08-18T11:10:47Z")

</div>

You'd need to have `negate => true`.

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 12:38pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/3 "2017-08-18T12:38:04Z")

</div>

We have already configured it to true but all the messages have been discarded. If we keep it to false we noticed the messages as single event per row.

Marcello

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 18, 2017, 12:52pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/4 "2017-08-18T12:52:32Z")

</div>

Then perhaps the match expression is wrong then. `negate => false` _will not work_.

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 12:59pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/5 "2017-08-18T12:59:11Z")

</div>

We configured the parameter and this is the full configuration:

```
input {
  syslog {
    type => "docker-dev-support"
    host => "0.0.0.0"
    port => 6000
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
    }
  }
}

filter {
  if [type] == "docker-dev-support" {
    if "_grokparsefailure_sysloginput" in [tags] {
      mutate {
        remove_tag => ["_grokparsefailure_sysloginput"]
      }
    }
    mutate {
      remove_field => ["message"]
    }
  }
}

output {
  if [type] == "docker-dev-support" {
   elasticsearch {
     action => "index"
     codec => "plain"
     hosts => ["http://localhost:9200"]
     index => "logstash-docker-test-dev-support-%{+YYYY.MM.dd}"
   }
  }
} 

```

In this scenario all the messages are discarded and we can't find them into an index (tagged as failed). We can't debug this configuration.

Marcello

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 18, 2017, 1:23pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/6 "2017-08-18T13:23:21Z")

</div>

Without knowing what your messages look like we can't help you debug it.

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 1:36pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/7 "2017-08-18T13:36:48Z")

</div>

The messages received are all forwarded via syslog from a logspout instance and we can have apache access log entries or exceptions like this:

```
2017-08-18 13:32:13,998 DEBUG [http-nio-8080-exec-6] testcharts [test.java:78] Error in test: 
java.lang.NullPointerException: null
        at it.company.mycards.view.test.doPost(test.java:61)
        at it.company.mycards.view.test.doGet(test.java:31)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 18, 2017, 1:47pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/8 "2017-08-18T13:47:01Z")

</div>

Yes, but that message is wrapped in a syslog payload isn't it? What does an actual syslog payload look like? I'm pretty sure the multiline codec is applied to the payload itself rather than the parsed message that the grok expression called internally by the syslog filter extracts.

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 2:06pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/9 "2017-08-18T14:06:39Z")

</div>

If i disable the negate to false and I keep alive the message for all the messages I noticed this scenario:

message: \<14\>1 2017-08-18T14:02:06Z 0fe037477a82 mycard-dev 25523 - - at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)

syslog\_message: at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)

It seems catched correctly from syslog input, but the multiline doesn't create entries into the index with the items catched, because I expect to see some grokparsefailure message in case of issue into the index.

Marcello

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 18, 2017, 2:27pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/10 "2017-08-18T14:27:01Z")

</div>

Right, that's what the real payload looks like. So your multiline configuration needs to take the syslog prefix into account and join with the previous line _unless_ it begins with something like this:

```
<14>1 2017-08-18T14:02:06Z 0fe037477a82 mycard-dev 25523 2017-08-18 13:32:13,998
```

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 3:06pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/11 "2017-08-18T15:06:14Z")

</div>

could it be fine with a configuration like this?

```
input {
  syslog {
    type => "docker-dev-support"
    host => "0.0.0.0"
    port => 6000
    codec => multiline {
      pattern => "^<%{NUMBER}>%{NUMBER} %{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
    }
  }
}

filter {
  if [type] == "docker-dev-support" {
    #grok {
    # patterns_dir => ["/etc/logstash/patterns/"]
    # match => { "message" => "%{DOCKERSYSLOG}" }
    #}
    #if "_grokparsefailure_sysloginput" in [tags] {
    # mutate {
    # remove_tag => ["_grokparsefailure_sysloginput"]
    # }
    #}
    #mutate {
    # remove_field => ["message"]
    #}
  }
}

output {
  if [type] == "docker-dev-support" {
   elasticsearch {
     action => "index"
     codec => "plain"
     hosts => ["http://localhost:9200"]
     index => "logstash-docker-test-dev-support-%{+YYYY.MM.dd}"
   }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 18, 2017, 4:22pm UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/12 "2017-08-18T16:22:41Z")

</div>

I think that this is the final configuration:

```
input {
  syslog {
    type => "docker-dev-support"
    host => "0.0.0.0"
    port => 6000
    codec => multiline {
      pattern => "^<%{NUMBER}>%{NUMBER} %{TIMESTAMP_ISO8601} %{NOTSPACE} %{NOTSPACE} %{NUMBER} - - (%{TIMESTAMP_ISO8601}|%{IPORHOST}|\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\])"
      negate => true
      what => "previous"
    }
  }
}

filter {
  if [type] == "docker-dev-support" {
    grok {
      patterns_dir => ["/etc/logstash/patterns/"]
      match => { "message" => "%{DOCKERSYSLOG}" }
    }
    if "_grokparsefailure_sysloginput" in [tags] {
      mutate {
        remove_tag => ["_grokparsefailure_sysloginput"]
      }
    }
    mutate {
      remove_field => ["message"]
    }
  }
}

```

output {  
if [type] == "docker-dev-support" {  
elasticsearch {  
action =\> "index"  
codec =\> "plain"  
hosts =\> ["[http://localhost:9200](http://localhost:9200)" ]  
index =\> "logstash-docker-test-dev-support-%{+YYYY.MM.dd}"  
}  
}  
}

I need to check if the logstash wait a new closing line to store the last rows processed into the multiline pipeline.

---

<div class="post-metadata">

**Author:** ![Marcello\_A](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcello_a/32/78018_2.png) [@Marcello\_A](https://discuss.elastic.co/u/Marcello_A)\
**Post date:** [August 21, 2017, 9:20am UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/13 "2017-08-21T09:20:18Z")

</div>

We have resolved during the weekend with this final configuration :

```
input {
  syslog {
    type => "docker-dev-support"
    host => "0.0.0.0"
    port => 6000
    codec => multiline {
      pattern => "^<%{NUMBER}>%{NUMBER} %{TIMESTAMP_ISO8601} %{NOTSPACE} %{NOTSPACE} %{NUMBER} - - (%{TIMESTAMP_ISO8601}|%{IPORHOST}|\[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\])"
      negate => true
      what => "previous"
    }
  }
}

filter {
  if [type] == "docker-dev-support" {
    grok {
      patterns_dir => ["/etc/logstash/patterns/"]
      match => { "message" => "%{DOCKERSYSLOG}" }
    }
    if "_grokparsefailure_sysloginput" in [tags] {
      mutate {
        remove_tag => ["_grokparsefailure_sysloginput"]
      }
    }
    if "multiline" in [tags] {
      mutate {
        gsub => ["syslog_message", "^<.*>.* \- \-", ""]
      } 
    }
    mutate {
      remove_field => ["message"]
    }
  }
}

output {
  if [type] == "docker-dev-support" {
   elasticsearch {
     action => "index"
     codec => "plain"
     hosts => ["http://localhost:9200"]
     index => "logstash-docker-test-dev-support-%{+YYYY.MM.dd}"    
   }
}

```

thanks for all the support,  
Marcello

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2017, 9:35am UTC](https://discuss.elastic.co/t/logstash-vs-rsyslog-java-stack-multiline-issue/97557/14 "2017-09-18T09:35:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.