# Forked Pipelines

**URL:** <https://discuss.elastic.co/t/forked-pipelines/217151>\
**Category:** Logstash\
**Created:** [January 30, 2020, 10:05am UTC](https://discuss.elastic.co/t/forked-pipelines/217151 "2020-01-30T10:05:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BeMoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bemoore/32/58724_2.png) [@BeMoore](https://discuss.elastic.co/u/BeMoore)\
**Post date:** [January 30, 2020, 10:06am UTC](https://discuss.elastic.co/t/forked-pipelines/217151/1 "2020-01-30T10:06:00Z")

</div>

Hi all,

I'm working on some forked pipelines and im getting a little confused on how this should be working... and this is sort of a continuation from this thread [Single data source split to different indexes](https://discuss.elastic.co/t/single-data-source-split-to-different-indexes/215695/4)

So the examples taken from [here](https://www.elastic.co/guide/en/logstash/master/pipeline-to-pipeline.html#collector-pattern)

```
pipeline.id: intake
  queue.type: persisted
  config.string: |
    input { beats { port => 5044 } }
    output { pipeline { send_to => ["internal-es", "partner-s3"] } }
- pipeline.id: buffered-es
  queue.type: persisted
  config.string: |
    input { pipeline { address => "internal-es" } }
    # Index the full event
    output { elasticsearch { } }
- pipeline.id: partner
  queue.type: persisted
  config.string: |
    input { pipeline { address => "partner-s3" } }
    filter {
      # Remove the sensitive data
      mutate { remove_field => 'sensitive-data' }
    }
    output { s3 { } } # Output to partner's bucket

```

So the issue thats confusing me is this:

config.string: |  
input { beats { port =\> 5044 } }  
output { pipeline { send\_to =\> ["internal-es", "partner-s3"] } }

specifically the input line.

i can't find anything that points me to what sort of inputs i can declare here. i'm trying the piplines.yml below and forking the data to two separate pipelines

```
- pipline.id: main 
  path.config: "C:\logstash-7.5.2\logstash-7.5.2\config\conf.d\syslog.conf"
  config.string: output { pipeline { send_to => [fortigate],[mswinevent] } }
  pipeline.workers: 1
  -pipline.id: fortigate 
  config.string: input { pipeline { address => fortigate } }
  path.config: "C:\logstash-7.5.2\logstash-7.5.2\config\conf.d\fortigate\fortigate.conf"
  pipeline.workers: 1
  - pipline.id: mswinevent
  config.string: input { pipeline { address => mswinevent } }
  path.config: "C:\logstash-7.5.2\logstash-7.5.2\config\conf.d\mswinevent\mswinevent.conf"
  pipeline.workers: 1

```

the logic to this is:  
I retain a full copy of all the data via the syslog inbound channel but its split to fortigate and mswinevents for further processing to separate indexes and will drop anything not applicable

main ---\> ES/syslog.conf  
|---\>fortigate---\> fortigate.conf -----\> ES/fortigate.index  
|---\> mswinevent----\> ES/mswinevent.index

I've been playing about with the inputs/outputs and the only thing that happens is that the syslog data is processed and nothing hits fortigate or mswinevents pipelines.

so in the syslog.conf

```
input {
  udp {
    port => 5000
    type => syslog
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["http://192.168.170.155:9200"]
	index => "syslog"
    pipeline => "fortigate"
	}
}

```

i've tried:

```
    pipeline => "[fortigate]"
    pipeline => "%{fortigate}"
    pipeline => "${[fortigate]}"
    pipeline => "fortigate"

```

none of this works..... is there and easier way of either splitting the feeds at the pipeline.yml stage or ingesting via the logstash syslog.cong, processing then spitting it out to ES and another pipeline?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 30, 2020, 4:37pm UTC](https://discuss.elastic.co/t/forked-pipelines/217151/2 "2020-01-30T16:37:24Z")

</div>

The pipeline option on an elasticsearch output refers to an elasticsearch ingestion pipeline, not to a logstash pipeline.

I have not tested whether you can have both a path.config and a config.string for a pipeline in pipelines.yml, but I would expect one or the other will be ignored.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2020, 4:37pm UTC](https://discuss.elastic.co/t/forked-pipelines/217151/3 "2020-02-27T16:37:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.