# Is there a way to direct different log formats from the same log to separate pipelines in filebeat?

**URL:** <https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 19, 2021, 10:36am UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565 "2021-04-19T10:36:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [April 19, 2021, 10:36am UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/1 "2021-04-19T10:36:24Z")

</div>

Hello,

I am having some logs coming from java app and the logs from jvm is not parsed by the java logger therefore it has a different format. For example:

```
{"@timestamp":"2021-04-19T10:25:51.985Z", "log.level": "INFO", "message":"Using [4] quartz threadCount.", "ecs.version": "1.2.0","service.name":"jetty","event.dataset":"jetty.log","process.thread.name":"main","log.logger":"org.quartz.SchedulerFactoryProvider"}
quartz threadCount = 42021-04-19T10:25:51.985385048Z
{"@timestamp":"2021-04-19T10:25:52.056Z", "log.level": "INFO", "message":"Using default implementation for ThreadExecutor", "ecs.version": "1.2.0","service.name":"jetty","event.dataset":"jetty.log","process.thread.name":"main","log.logger":"org.quartz.impl.StdSchedulerFactory"}

```

I want the json format to be sent to the elasticsearch directly while the quartz log to be parsed by some logstash pipeline. Is there something that can help me achieve the above?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 22, 2021, 1:24am UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/4 "2021-04-22T01:24:33Z")

</div>

Sonu can only have one output but instead of using logstash pipelines, u could send it all to elasticsearch and use elasticsearch pipelines and u can do sub-pipelines bases on conditionals like if the first character in the log is { for json logs or not.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 22, 2021, 1:24am UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/5 "2021-04-22T01:24:57Z")

</div>

Or send all to logstash and do the same type of conditional processing

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [April 22, 2021, 8:08am UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/6 "2021-04-22T08:08:33Z")

</div>

let's take it one step further. The logs are java logs and usually that are multiline. Most of them are caught by ecs.logger and formatted in json properly but sometimes we have exception in thread which are multiline and I want them to be handled as a single doc not 3,4,5,6... different documents. Any idea how to go about that?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 23, 2021, 2:13pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/7 "2021-04-23T14:13:05Z")

</div>

You can setup multi line logging, see [Manage multiline messages | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2021, 4:13pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-direct-different-log-formats-from-the-same-log-to-separate-pipelines-in-filebeat/270565/8 "2021-05-21T16:13:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.