Combine filebeat logs to ship to Elastic Search based on unique trace id (without Logstash)

Elastic Stack Beats
1

I have configured filebeat for shipping logs of my spring boot application directly to Elasticsearch without configuration of logstash. I want to configure filebeat multiline regex in a way that it can combine all the logs in a single record having same transactionId. Can anyone please suggest ? Here the part of log configuration looks like in filebeat.yml:

filebeat.inputs:
- type: log
  paths:
    - E://filebeat//*.log
  reload.enabled: true
  reload.period: 300s
  enabled: true
  multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after
     
Log Sample:

2022-03-07 07:08:44 [ERROR] | transactionId=6af42925-48df-4f49-95a4-aa0e43b152ed | c.example.utility.advice.LoggingAdvice - Method Signature:  Boolean com.example.config.service.impl.ServiceImpl.testLogger() - Line No : 1345 - Exception :  null
2022-03-07 07:08:50 [ERROR] | transactionId=51a2d445-574b-4992-b867-a7fc5ee4b473 | c.example.utility.advice.LoggingAdvice - Method Signature:  Boolean com.example.config.service.impl.ServiceImpl.testLogger() - Line No : 1345 - Exception :  null
2022-03-07 05:26:08 [ERROR] | transactionId=17f4f815-ee74-4658-a8d7-347fea6dd9d1 | o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
    at com.example.config.service.impl.ServiceImpl.testLogger(EventServiceImpl.java:1345) ~[classes/:na]
    at com.example.config.service.impl.ServiceImpl$FastClassBySpringCGLIB$983a4fb2.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.9.jar:5.3.9]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.9.jar:5.3.9]
    at
3

Hi @shikha1!
I think your multiline pattern is not correct. Could you try this configuration:

  multiline:
    type: pattern
    pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
4

I think that the multiline pattern should be just:

pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

There is no [ to escape, this pattern will get all lines that start with a date in the format yyyy-MM-dd.

5

Thanks Leandrojmp and Tetiana. This pattern seems to be working fine for me when I want to combine exception logs in my java application. However, I want to combine logs based on unique words such as trace id or transaction id in single records along with these exception logs. Is there any suggestion on that you can provide please ?

6

That's not possible with filebeat alone, if you want to aggregate lines based on some specific ID you will need to use Logstash to do that.

Logstash has the aggregate filter to join lines based on a ID.

You would need to send your logs to logstash and then to Elasticsearch.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.