Pushing different types of (access, application) logs using filebeat-daemonset in Kubernetes

I have several pods running spring-boot applications, writing application-logs to stdout, which is then picked up by filebeat-daemonset and then pushed to elasticsearch. (standard logging architecture for kubernetes)

Now, I need to enable tomcat-access-logs in spring-boot applications. Should I enable it and make them write to stdout as well? So the same container would write two types of logs to stdout . The issue would be, both (application, access) logs will have different patterns. I need to dissect them differently so that in kibana, I can setup indices based on them.

With filebeat, can we configure different log-patterns (and then dissect) from the same source ( stdout , in this case)? Are there any known best practices for this requirement? In VM-based deployments, I used to have more than one filebeat.yml files for a single filebeat, which were reading log-files from different sources (/var/log/application*.log, /var/log/access*.log).

I have asked same question on Stackoverflow (with sample code) if you could please help.

The common practice is to involve the ingest node pipeline. See this thread: Best practice for Logging with Filebeat

Thanks @mtojek for the recommendations. Can you please add more info on how ingest node pipeline would help in my scenario?

Apologies, as I am not able to visualize the solution here.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.