Java multiline stack trace log shipping with filebeat

Elastic Stack Beats
1

Hi, Please help me for creating a pattern for this log

[2022-05-07 13:20:53,621] [WARN] [gateway_service] [reactor.util.Loggers$Slf4JLogger] [warn:295] message: [b17d8e8c-1, L:/192.168.149.185:33094 - R:orderscl.sepanta.svc.cluster.local/10.97.164.58:9010] The connection observed an error
io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 403 Forbidden
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272)
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:304)
        at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:116)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

It doesn't get all of this lines as a single event.
thanks in advance

2

There is an example in Filebeat documentation Manage multiline messages | Filebeat Reference [8.2] | Elastic.

3

Thanks, I tried them but it didn't work.

4

Hi @Alireza_Zabihi Welcome to the community.

What I suggest if you would like help is to

  1. Provide 3 log entries
    1st entry normal
    2nd entry the multi-line stack trace
    3rd entry a normal lin

So we can understand the pattern.

  1. Provide your filebeat config to see what you have tried, and what your configuration is.

Then perhaps we can help a bit more.

1 Like
5

Hi @stephenb , Thanks you so much. this is logs that i have in my kubernetes pod output:

[2022-05-09 09:45:45,916] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"fc319cc2-2011","User-IP":"10.0.2.2","URI":"GET http://10.0.2.100:8082/orders"}
[2022-05-09 09:45:45,941] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"4ebecfb0-2012","User-IP":"10.0.2.2","URI":"GET http://10.0.2.100:8082/market-data/socket"}
[2022-05-09 09:46:21,457] [ERROR] [gateway_service] [reactor.util.Loggers$Slf4JLogger] [error:315] message: Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalArgumentException: WebSocket close status code does NOT comply with RFC-6455: 1005
Caused by: java.lang.IllegalArgumentException: WebSocket close status code does NOT comply with RFC-6455: 1005
        at io.netty.handler.codec.http.websocketx.CloseWebSocketFrame.requireValidStatusCode(CloseWebSocketFrame.java:209)
        at io.netty.handler.codec.http.websocketx.CloseWebSocketFrame.<init>(CloseWebSocketFrame.java:69)
        at reactor.netty.http.client.WebsocketClientOperations.sendClose(WebsocketClientOperations.java:215)
        at org.springframework.web.reactive.socket.adapter.ReactorNettyWebSocketSession.close(ReactorNettyWebSocketSession.java:124)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
        at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
        at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
        at reactor.core.publisher.FluxFirstWithSignal$FirstEmittingSubscriber.onNext(FluxFirstWithSignal.java:330)
        at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2634)
        at reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:70)
        at reactor.netty.http.server.WebsocketServerOperations.sendCloseNow(WebsocketServerOperations.java:260)
        at reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:158)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2022-05-09 09:46:21,465] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.keycloak.KeycloakWebClientService] [lambda$introspect$2:125] message: d~~~~~~~~~~~~~~~~~~~~~~~~~~ > User authentication status:  {"auth_status":true,"token_abr":{"start":"eyJhb","end":"x87Dw"},"userId":"58ed3a07-230b-4550-8ca8-8dc4f2152744","sessionState":"40a6f1e2-9e09-4449-93f9-bd8f60920629","username":"frontuser"}
[2022-05-09 09:46:21,466] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"48622a31-2013","User-IP":"10.0.2.2","URI":"GET http://10.0.0.0:8050/market-data/instruments/instrument-sector/27?page=1&size=10"}
[2022-05-09 09:46:21,554] [DEBUG] [gateway_service] [com.sepantasolutions.maingateway.configurations.interceptors.RemoteAddressKeyResolver] [resolve:31] message: d~~~~~~~~~~~~~~~~~~~~~~~~~~ > Request detail: {"RequestId":"ea656e1d-2014","User-IP":"10.0.2.2","URI":"GET http://10.0.0.0:8050/market-data/socket"}

and this is my kubernetes configmap for filebeat:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elk
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          multiline.type: pattern
          multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
          multiline.negate: false
          multiline.match: after
    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elkstack-coordinating-only}:${ELASTICSEARCH_PORT:9200}']
      protocol: "https"
      ssl.verification_mode: none
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

Thanks in advance dear @stephenb

6

Hi @Alireza_Zabihi
Assuming you are using the new filestreams this worked for me... note the new parsers configuration field (basically your multiline is just being ignored) Yup the docs should make that clearer...

This is just a normal filebeat.yml so you just need to translate that into your configmap

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/filebeat-multiline/java-except.log
    # - /Users/sbrown/workspace/sample-data/spring-boot-log/spring-short.log
    # - /Users/sbrown/workspace/sample-data/nginx/nginx2020.log
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

    parsers:
      - multiline:
          type: pattern
          pattern: '^\['
          negate: true
          match: after

Results

Screen Shot 2022-05-09 at 7.40.39 AM
Screen Shot 2022-05-09 at 7.40.39 AM2978×1012 339 KB

Probably should look something like

          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            parsers:
              - multiline:
                  type: pattern
                  pattern: '^\['
                  negate: true
                  match: after
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.