Fleet agent custom log integration multiline settings

Elastic Stack Beats
1

Hello all,
I try to setup a custom log policy for our java logs. Previously we used filebeat and had the following lines in filebeat.yml to configure multiline log entries:

  multiline:
    match: after
    negate: true
    pattern: ^[0-9]+

This resulted having logs like the following in one document containing the full stack trace.

2022-02-14T09:56:58,141 c2_hw_nav int  qtp379303133-267167 WARN HttpChannel - /babelfish/display/v1/barwithstopindication
javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: Duplicate key apricot (attempted merging values ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@29750737[type=BAR,address=9,codepageId=apricot,producer=GORBA,trailer=false] and ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@485e5b3a[type=BAR,address=6,codepageId=apricot,producer=GORBA,trailer=false])
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]

Now with elastic-agent I installed the custom log integration and created a policy to achieve the same. But this does not seem to work.
The relevant configuration on the agent looks as follows:

- data_stream:
    namespace: vehicle
  id: 2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
  meta:
    package:
      name: log
      version: 1.0.0
  name: custom-logs
  revision: 36
  streams:
  - data_stream:
      dataset: vehicle-service
    exclude_files:
    - passengerCounter.log$
    id: logfile-log.log-2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
    multiline:
      match: after
      negate: true
      pattern: ^[0-9]{4}
    paths:
    - /usr/share/elastic-agent/logs/c*/*.log
    - /usr/share/elastic-agent/logs/eagle/eagle.log
    pipeline: isa-vehicle-service-logs
  type: logfile
  use_output: default

In the agent logs I cannot find any errors. The message arrives in Elasticsearch but the multiline part is missing.
Is there anything I did wrong?

2

I finally managed to solve it by my self. There was an error in the Grok pattern of the ingest pipeline.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.