Is there a way to direct different log formats from the same log to separate pipelines in filebeat?

Elastic Stack Beats
1

Hello,

I am having some logs coming from java app and the logs from jvm is not parsed by the java logger therefore it has a different format. For example:

{"@timestamp":"2021-04-19T10:25:51.985Z", "log.level": "INFO", "message":"Using [4] quartz threadCount.", "ecs.version": "1.2.0","service.name":"jetty","event.dataset":"jetty.log","process.thread.name":"main","log.logger":"org.quartz.SchedulerFactoryProvider"}
quartz threadCount = 42021-04-19T10:25:51.985385048Z
{"@timestamp":"2021-04-19T10:25:52.056Z", "log.level": "INFO", "message":"Using default implementation for ThreadExecutor", "ecs.version": "1.2.0","service.name":"jetty","event.dataset":"jetty.log","process.thread.name":"main","log.logger":"org.quartz.impl.StdSchedulerFactory"}

I want the json format to be sent to the elasticsearch directly while the quartz log to be parsed by some logstash pipeline. Is there something that can help me achieve the above?

4

Sonu can only have one output but instead of using logstash pipelines, u could send it all to elasticsearch and use elasticsearch pipelines and u can do sub-pipelines bases on conditionals like if the first character in the log is { for json logs or not.

5

Or send all to logstash and do the same type of conditional processing

6

let's take it one step further. The logs are java logs and usually that are multiline. Most of them are caught by ecs.logger and formatted in json properly but sometimes we have exception in thread which are multiline and I want them to be handled as a single doc not 3,4,5,6... different documents. Any idea how to go about that?

7

You can setup multi line logging, see Manage multiline messages | Filebeat Reference [7.12] | Elastic

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.