Fleet agent custom log integration multiline settings

Hello all,
I try to setup a custom log policy for our java logs. Previously we used filebeat and had the following lines in filebeat.yml to configure multiline log entries:

    match: after
    negate: true
    pattern: ^[0-9]+

This resulted having logs like the following in one document containing the full stack trace.

2022-02-14T09:56:58,141 c2_hw_nav int  qtp379303133-267167 WARN HttpChannel - /babelfish/display/v1/barwithstopindication
javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: Duplicate key apricot (attempted merging values ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@29750737[type=BAR,address=9,codepageId=apricot,producer=GORBA,trailer=false] and ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@485e5b3a[type=BAR,address=6,codepageId=apricot,producer=GORBA,trailer=false])
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]

Now with elastic-agent I installed the custom log integration and created a policy to achieve the same. But this does not seem to work.
The relevant configuration on the agent looks as follows:

- data_stream:
    namespace: vehicle
  id: 2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
      name: log
      version: 1.0.0
  name: custom-logs
  revision: 36
  - data_stream:
      dataset: vehicle-service
    - passengerCounter.log$
    id: logfile-log.log-2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
      match: after
      negate: true
      pattern: ^[0-9]{4}
    - /usr/share/elastic-agent/logs/c*/*.log
    - /usr/share/elastic-agent/logs/eagle/eagle.log
    pipeline: isa-vehicle-service-logs
  type: logfile
  use_output: default

In the agent logs I cannot find any errors. The message arrives in Elasticsearch but the multiline part is missing.
Is there anything I did wrong?

I finally managed to solve it by my self. There was an error in the Grok pattern of the ingest pipeline.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.