Hello all,
I try to setup a custom log policy for our java logs. Previously we used filebeat and had the following lines in filebeat.yml to configure multiline log entries:
multiline:
match: after
negate: true
pattern: ^[0-9]+
This resulted having logs like the following in one document containing the full stack trace.
2022-02-14T09:56:58,141 c2_hw_nav int qtp379303133-267167 WARN HttpChannel - /babelfish/display/v1/barwithstopindication
javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: Duplicate key apricot (attempted merging values ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@29750737[type=BAR,address=9,codepageId=apricot,producer=GORBA,trailer=false] and ch.post.it.paisa.api.vehicle.setup.v1.IbisDisplayDevice@485e5b3a[type=BAR,address=6,codepageId=apricot,producer=GORBA,trailer=false])
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
Now with elastic-agent I installed the custom log integration and created a policy to achieve the same. But this does not seem to work.
The relevant configuration on the agent looks as follows:
- data_stream:
namespace: vehicle
id: 2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
meta:
package:
name: log
version: 1.0.0
name: custom-logs
revision: 36
streams:
- data_stream:
dataset: vehicle-service
exclude_files:
- passengerCounter.log$
id: logfile-log.log-2f1c84e8-45bf-4106-aee2-3fc5e8f19af8
multiline:
match: after
negate: true
pattern: ^[0-9]{4}
paths:
- /usr/share/elastic-agent/logs/c*/*.log
- /usr/share/elastic-agent/logs/eagle/eagle.log
pipeline: isa-vehicle-service-logs
type: logfile
use_output: default
In the agent logs I cannot find any errors. The message arrives in Elasticsearch but the multiline part is missing.
Is there anything I did wrong?