# Java multiline stack trace log shipping with filebeat

**URL:** <https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 7, 2022, 8:56am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170 "2022-05-07T08:56:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alireza\_Zabihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alireza_zabihi/32/105374_2.png) [@Alireza\_Zabihi](https://discuss.elastic.co/u/Alireza_Zabihi)\
**Post date:** [May 7, 2022, 8:56am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/1 "2022-05-07T08:56:07Z")

</div>

Hi, Please help me for creating a pattern for this log

```auto
[2022-05-07 13:20:53,621] [WARN] [gateway_service] [reactor.util.Loggers$Slf4JLogger] [warn:295] message: [b17d8e8c-1, L:/192.168.149.185:33094 - R:orderscl.sepanta.svc.cluster.local/10.97.164.58:9010] The connection observed an error
io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 403 Forbidden
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272)
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:304)
        at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:116)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

```

* * *

It doesn't get all of this lines as a single event.  
thanks in advance

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [May 7, 2022, 9:41am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/2 "2022-05-07T09:41:52Z")

</div>

There is an example in Filebeat documentation [Manage multiline messages | Filebeat Reference [8.2] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html).

---

<div class="post-metadata">

**Author:** ![Alireza\_Zabihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alireza_zabihi/32/105374_2.png) [@Alireza\_Zabihi](https://discuss.elastic.co/u/Alireza_Zabihi)\
**Post date:** [May 7, 2022, 9:52am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/3 "2022-05-07T09:52:00Z")

</div>

Thanks, I tried them but it didn't work.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 8, 2022, 12:58am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/4 "2022-05-08T00:58:14Z")

</div>

Hi @Alireza_Zabihi Welcome to the community.

What I suggest if you would like help is to

1. Provide 3 log entries  
1st entry normal  
2nd entry the multi-line stack trace  
3rd entry a normal lin

So we can understand the pattern.

1. Provide your filebeat config to see what you have tried, and what your configuration is.

Then perhaps we can help a bit more.

---

<div class="post-metadata">

**Author:** ![Alireza\_Zabihi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alireza_zabihi/32/105374_2.png) [@Alireza\_Zabihi](https://discuss.elastic.co/u/Alireza_Zabihi)\
**Post date:** [May 9, 2022, 10:00am UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/5 "2022-05-09T10:00:23Z")

</div>

Hi @stephenb , Thanks you so much. this is logs that i have in my kubernetes pod output:

```auto
[2022-05-09 09:45:45,916] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d ~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"fc319cc2-2011","User-IP":"10.0.2.2","URI":"GET http://10.0.2.100:8082/orders"}
[2022-05-09 09:45:45,941] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d ~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"4ebecfb0-2012","User-IP":"10.0.2.2","URI":"GET http://10.0.2.100:8082/market-data/socket"}
[2022-05-09 09:46:21,457] [ERROR] [gateway_service] [reactor.util.Loggers$Slf4JLogger] [error:315] message: Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalArgumentException: WebSocket close status code does NOT comply with RFC-6455: 1005
Caused by: java.lang.IllegalArgumentException: WebSocket close status code does NOT comply with RFC-6455: 1005
        at io.netty.handler.codec.http.websocketx.CloseWebSocketFrame.requireValidStatusCode(CloseWebSocketFrame.java:209)
        at io.netty.handler.codec.http.websocketx.CloseWebSocketFrame.<init>(CloseWebSocketFrame.java:69)
        at reactor.netty.http.client.WebsocketClientOperations.sendClose(WebsocketClientOperations.java:215)
        at org.springframework.web.reactive.socket.adapter.ReactorNettyWebSocketSession.close(ReactorNettyWebSocketSession.java:124)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
        at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
        at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
        at reactor.core.publisher.FluxFirstWithSignal$FirstEmittingSubscriber.onNext(FluxFirstWithSignal.java:330)
        at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2634)
        at reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:70)
        at reactor.netty.http.server.WebsocketServerOperations.sendCloseNow(WebsocketServerOperations.java:260)
        at reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:158)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2022-05-09 09:46:21,465] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.keycloak.KeycloakWebClientService] [lambda$introspect$2:125] message: d ~~~~~~~~~~~~~~~~~~~~~~~~~~ > User authentication status: {"auth_status":true,"token_abr":{"start":"eyJhb","end":"x87Dw"},"userId":"58ed3a07-230b-4550-8ca8-8dc4f2152744","sessionState":"40a6f1e2-9e09-4449-93f9-bd8f60920629","username":"frontuser"}
[2022-05-09 09:46:21,466] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d ~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"48622a31-2013","User-IP":"10.0.2.2","URI":"GET http://10.0.0.0:8050/market-data/instruments/instrument-sector/27?page=1&size=10"}
[2022-05-09 09:46:21,554] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d ~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"ea656e1d-2014","User-IP":"10.0.2.2","URI":"GET http://10.0.0.0:8050/market-data/socket"}

```

and this is my kubernetes configmap for filebeat:

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elk
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          multiline.type: pattern
          multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
          multiline.negate: false
          multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elkstack-coordinating-only}:${ELASTICSEARCH_PORT:9200}']
      protocol: "https"
      ssl.verification_mode: none
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

```

Thanks in advance dear @stephenb

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 9, 2022, 2:43pm UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/6 "2022-05-09T14:43:56Z")

</div>

Hi @Alireza_Zabihi  
Assuming you are using the new filestreams this worked for me... **note the new** [`parsers`](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html) configuration field (basically your multiline is just being ignored) Yup the docs should make that clearer...

This is just a normal filebeat.yml so you just need to translate that into your configmap

```auto
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/java-except.log
    # - /Users/sbrown/workspace/sample-data/spring-boot-log/spring-short.log
    # - /Users/sbrown/workspace/sample-data/nginx/nginx2020.log
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

    parsers:
      - multiline:
          type: pattern
          pattern: '^\['
          negate: true
          match: after

```

Results

 ![Screen Shot 2022-05-09 at 7.40.39 AM](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d26d88ceba86f64c64ea5e53bdcd5ea62b5c8254.png)

Probably should look something like

```auto
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            parsers:
              - multiline:
                  type: pattern
                  pattern: '^\['
                  negate: true
                  match: after

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 6, 2022, 4:44pm UTC](https://discuss.elastic.co/t/java-multiline-stack-trace-log-shipping-with-filebeat/304170/7 "2022-06-06T16:44:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.