# Grokparse failure for multiline ERROR messages but not for WARN log levels

**URL:** <https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252>\
**Category:** Logstash\
**Created:** [June 30, 2020, 8:33am UTC](https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252 "2020-06-30T08:33:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ramokarahasan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramokarahasan/32/66226_2.png) [@ramokarahasan](https://discuss.elastic.co/u/ramokarahasan)\
**Post date:** [June 30, 2020, 8:33am UTC](https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252/1 "2020-06-30T08:33:56Z")

</div>

Hi,

I'm using filebeat -\> logstash -\> elasticsearch -\> kibana and would like to have the loglevels displayed in kibana.

Filebeat 7.6.2  
Logstash 7.6.1

This works fine with WARN and INFO but not for ERROR log levels.

My multiline filebeat config looks like

```auto
filebeat.inputs: # List of inputs.
- type: log
  enabled: true
  paths:
    - /srv/iws/logs/*/*.log
  multiline.pattern: '^\d\d\d\d-\d\d-\d\d'
  multiline.negate: true
  multiline.match: after

```

the logstash config look

```auto
filter {

  mutate {
     remove_field => ["host"]
  }

  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }

  grok {
    match => [ "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<message>.*)"
             ]
  }

}

```

and the error messages that are throwing grokparse failures are the following

```auto
2020-06-30 07:47:51.853 ERROR 1 --- [elastic-19629] s.ConnectorConfigServiceExceptionHandler : Exception handled

com.itembase.iws.connector.config.ConnectorConfigNotFoundException: Cannot find configuration for connectionId 1cfc914e-e37e-4528-8fcc-f0965079b8c2
	at com.itembase.iws.connector.config.webservice.connection.ConnectionEntityService.getById(ConnectionEntityService.java:75) ~[classes/:na]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler com.itembase.iws.connector.config.webservice.connection.ConnectionRestController#get(UUID) [DispatcherHandler]
Stack trace:
		at com.itembase.iws.connector.config.webservice.connection.ConnectionEntityService.getById(ConnectionEntityService.java:75) ~[classes/:na]
		at com.itembase.iws.connector.config.webservice.connection.ConnectionRestController.get(ConnectionRestController.java:46) ~[classes/:na]
		at jdk.internal.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) ~[na:na]
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
		at org.springframework.web.reactive.result.method.InvocableHandlerMethod.lambda$invoke$0(InvocableHandlerMethod.java:147) ~[spring-webflux-5.2.4.RELEASE.jar:5.2.4.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1705) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2267) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2075) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1949) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:191) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4095) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2267) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2075) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1949) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4110) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4110) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.3.RELEASE.jar:3.3.3.RELEASE]
		at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64) ~[reactor-netty-0.9.5.RELEASE.jar:0.9.5.RELEASE]
		at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:228) ~[reactor-netty-0.9.5.RELEASE.jar:0.9.5.RELEASE]
		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:465) ~[reactor-netty-0.9.5.RELEASE.jar:0.9.5.RELEASE]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90) ~[reactor-netty-0.9.5.RELEASE.jar:0.9.5.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:170) ~[reactor-netty-0.9.5.RELEASE.jar:0.9.5.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		

```

I'm not sure if this is the multiline plugin since WARN and INFO messages are not multiline

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 30, 2020, 3:51pm UTC](https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252/2 "2020-06-30T15:51:37Z")

</div>

Which grok is failing? You can test that using add\_field for each one.

Also, you appear to have two spaces before LOGLEVEL but only one in your message.

---

<div class="post-metadata">

**Author:** ![ramokarahasan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramokarahasan/32/66226_2.png) [@ramokarahasan](https://discuss.elastic.co/u/ramokarahasan)\
**Post date:** [July 2, 2020, 9:53am UTC](https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252/3 "2020-07-02T09:53:06Z")

</div>

Thanks @Badger ,

indeed the patterns didn't support a flexible whitespace detection. Fixed it by adding `\s*` infront of the loglevel pattern.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2020, 9:53am UTC](https://discuss.elastic.co/t/grokparse-failure-for-multiline-error-messages-but-not-for-warn-log-levels/239252/4 "2020-07-30T09:53:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.