Working with multiline messages on Elastic Agent

Elastic Stack Elastic Agent
1

Hello fellow devs, i have a usecase to ingest a application log to the elastic using Elastic Agent on my Java application, right now I got a problem when the application caught an unhandled exception and it prints it to the server log. My goals is to make the multi line exception message into single event.

Exception sample:

2024-05-06 14:46:22 ICT [SCC.0126.0200I] (tid=351) SCC ConnectionManager pool KomiUBPJDBCConn.conn:KomiUBPNoTrx started 
2024-05-06 14:46:45 ICT [ART.0114.1100I] (tid=351) Adapter Runtime: Facility 1 - JDBCAdapter registered with bundle com.wm.adapter.wmjdbc.JDBCAdapterResourceBundle. 
2024-05-06 14:46:45 ICT [ISS.0095.0042I] (tid=351) The ERRSTACKTRACE field in a WMERROR audit record was truncated. CONTEXTID: ee93ae3f-59a4-4af7-a2ee-70a22cfdaad5. MSGID: 491d55b6-e8d6-f612-d8ea-608365a3fe29. Full value: java.io.IOException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
	at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:353)
	at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
	at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
	at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:252)
	at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:118)
	at oracle.net.ns.NSProtocol.connect(NSProtocol.java:317)
	at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518)

I tried to use multiline parser based on the documentation on my elastic-agent.yml but it still printing each line as single events

elastic-agent.yml inputs

inputs:
  - id: bni_app_logs
    type: filestream
    multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
      negate: true
      match: after
    paths:
      - /opt/softwareag/sag1015/IntegrationServer/instances/is5555/logs/server.log
    data_stream:
      namespace: bni_app_logs

If anyone had any opinion or suggestion, please help.
Thank You.

2

Hi,

you can test the config here:

Regards

3

I'm sorry, but i was looking the same reference before you directed me to the source, and the source showed that the parsers only works in filebeat, i tried applying it before and the elastic-agent won't even start, do you have any example file of the implementation?

4

Hi,

maybe this article can help you:

How to integrate custom logs with Elastic Agent | by Benoit Luttringer | Zenika

image
image702×699 82.4 KB

Regards