# Two pipelines for the same file **URL:** \ **Category:** Beats\ **Created:** [June 26, 2017, 8:40pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872 "2017-06-26T20:40:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hding](https://avatars.discourse-cdn.com/v4/letter/h/eb9ed0/32.png) [@hding](https://discuss.elastic.co/u/hding)\ **Post date:** [June 26, 2017, 8:40pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/1 "2017-06-26T20:40:57Z")
My log file contains log lines of two types: "NORMAL" and "ERROR". Each line starts with its type. The logs are stored in the same directory. I have built two pipelines: "pipeline\_1" and "pipeline\_2". How do I configure filebeat.prospectors and output.elasticsearch? Will the following work? ```auto filebeat.prospectors: - paths: ["/var/log/*.log"] include_lines: ["^NORMAL"] fields: type: "normal" - paths: ["/var/log/*.log"] include_lines: ["^ERROR"] fields: type: "error" output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: pipeline_1 when.equals: fields.type: "normal" - pipeline: pipeline_2 when.equals: fields.type: "error" ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 27, 2017, 8:31am UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/2 "2017-06-27T08:31:50Z")
Do not have any two prospectors report the same file. The files state/offset is currently stored global and these two prospectors will interleave. You can use the `when.equals` condition with `match` or `regex`, to check a log message starts with any of these, and then ship do to the respective pipeline. Are the logs that different, that you really need a different pipeline? e.g. in on can configure multiple patterns in the grok processor. ---
**Author:** ![hding](https://avatars.discourse-cdn.com/v4/letter/h/eb9ed0/32.png) [@hding](https://discuss.elastic.co/u/hding)\ **Post date:** [June 27, 2017, 12:40pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/3 "2017-06-27T12:40:48Z")
Thanks a lot, Steffen. By saying use `when.equals` with `match` and `regex`, do you mean like this: ```auto output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: pipeline_1 when.equals: regex: ["^NORMAL"] - pipeline: pipeline_2 when.equals: regex: ["^ERROR"] ``` The lines in one file can be indeed very different. One line type can have 80 fields while another can have only 7 fields. And there are over 20 different types... The first few letters in a line indicates its type. Right now I am using ingest node instead of Logstash. Will using multi-patterns in grok processor slow down the ingestion speed? Does grok try to match the patterns one by one until one is matched? Should I actually consider using Logstash with Filebeat? ---
**Author:** ![hding](https://avatars.discourse-cdn.com/v4/letter/h/eb9ed0/32.png) [@hding](https://discuss.elastic.co/u/hding)\ **Post date:** [June 27, 2017, 9:35pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/4 "2017-06-27T21:35:23Z")
I tried the following, but it doesn't work... ```auto # Configure indices indices: - index: "my_index_1" when.equals: match: ["^NORMAL"] - index: "my_index_2" when.equals: match: ["^ERROR"] # Configure pipelines # pipeline: "my_pipeline_r" pipelines: - pipeline: "my_pipeline_1" when.equals: match: ["^NORMAL"] - pipeline: "my_pipeline_2" when.equals: match: ["^ERROR"] ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 28, 2017, 10:19am UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/5 "2017-06-28T10:19:06Z")
> Right now I am using ingest node instead of Logstash. Will using multi-patterns in grok processor slow down the ingestion speed? No idea how grok is executed in Elasticsearch Ingest Node. But I'd assume it's one by one. Aynway, you having just 2 patterns shouldn't be much of a concern. Which filebeat version (match was introduced somewhat later, use regex )? Any logs? The syntax for the when clause is `when..: `. You have used `when..: `. Both, `equals` and `match` are conditions. Here is an example on [pipelines](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_pipelines). And [documentation on conditions](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-processors.html#conditions). Try: ```auto pipelines: - pipeline: "my_pipeline_1" when.regexp.message: '^NORMAL' - pipeline: "my_pipeline_2" when.regexp.message: '^ERROR'' ``` Do not use double quotes for regular expressions in YAML. ---
**Author:** ![hding](https://avatars.discourse-cdn.com/v4/letter/h/eb9ed0/32.png) [@hding](https://discuss.elastic.co/u/hding)\ **Post date:** [June 28, 2017, 12:51pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/6 "2017-06-28T12:51:57Z")
Thank you so much for your help, Steffen! It works like a magic! I'm using Filebeat 5.4.1 and ElasticSearch 5.4.1. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2017, 8:55pm UTC](https://discuss.elastic.co/t/two-pipelines-for-the-same-file/90872/7 "2017-07-17T20:55:58Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.