# Problem with "cloning" (filebeat -\> logstash -\> elasticsearch) - more logstash pipelines

**URL:** <https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018>\
**Category:** Logstash\
**Created:** [August 6, 2020, 12:05pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018 "2020-08-06T12:05:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rysiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rysiu/32/61919_2.png) [@Rysiu](https://discuss.elastic.co/u/Rysiu)\
**Post date:** [August 6, 2020, 12:05pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/1 "2020-08-06T12:05:10Z")

</div>

Hi,

I have a problem.  
In simple terms, the problem looks like this:

If I add one pipeline in logstash:

```
input {
   beats {
      ...
   }
}

filter {
...
}

output {
   elasticsearch {
      ...
   }
}

```

Everything works very well.

But if I use the configuration that we have one pipeline that distributes data from filebeat to two ports (in localhost):

```
input {
   beats {
      ...
   }
}

filter {
}

output {
   udp {
      host => "localhost"
      port => X
      codec => "json_lines"
   }
   udp {
      host => "localhost"
      port => Y
      codec => "json_lines"
   }
}

```

And then another pipeline receives/parse data from the port and adds it to elasticsearch:

```
input {
       udp {
          port => X
          codec => "json_lines"
       }
}

filter {
...
}

output {
   elasticsearch {
      ...
   }
}

```

Unfortunately, it's not working.  
So to speak quickly... I need to split one beats input into two separate pipeline in logstash.

I've read about pipeline-to-pipeline capabilities and lumbejack input/output.

But is there a possibility here of a simple forward after tcp/udp and such cloning?

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [August 6, 2020, 12:26pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/2 "2020-08-06T12:26:37Z")

</div>

We run several parallel Logstash pipelines

We have based our config on [this](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) documentation

I find the documentation pretty clear so I will let it speak on its own. I'm happy to try to answer questions if there are any.

---

<div class="post-metadata">

**Author:** ![Rysiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rysiu/32/61919_2.png) [@Rysiu](https://discuss.elastic.co/u/Rysiu)\
**Post date:** [August 6, 2020, 12:33pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/3 "2020-08-06T12:33:33Z")

</div>

Yes, but I must have two pipelines using data from the same network port (beats).  
Normally I also have several different pipelines in pipelines.yml.  
No problem and it works well but I always have one input to one pipeline.  
I need one input to two pipelines for some reasons.

This is not a standard configuration.  
However I want to have two different pipelines performing completely different parsing.  
It should also be assumed that I don't have the ability to add another output to filebeat and I want a very simple and fast solution on logstash.  
I guess we can do this on udp/tcp forwarding?

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [August 6, 2020, 1:06pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/4 "2020-08-06T13:06:26Z")

</div>

This is a completely theoretical, not tested, suggestion as I have not used pipeline-to-pipeline communication

I would expect something like this to do the job

```
# config/pipelines.yml
- pipeline.id: upstream
  config.string: input { beats {} } output { pipeline { send_to => [myVirtualAddress1] } pipeline { send_to => [myVirtualAddress2] } }
- pipeline.id: downstream1
  config.string: input { pipeline { address => myVirtualAddress1 } }
- pipeline.id: downstream2
    config.string: input { pipeline { address => myVirtualAddress2 } } 

```

I'm sure you would put the config for each pipeline in their own files as the lines would get hard to read otherwise but just to illustrate the main config.

---

<div class="post-metadata">

**Author:** ![Rysiu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rysiu/32/61919_2.png) [@Rysiu](https://discuss.elastic.co/u/Rysiu)\
**Post date:** [August 6, 2020, 1:14pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/5 "2020-08-06T13:14:59Z")

</div>

Yes.  
I tested it.  
But I have very extensive parsers described in the separated files.  
One pipeline is one quite long file with input/filter/output (filters can be quite a lot).

Such configuration probably works only if everything is in one config/pipelines.yml file.

But when I was adding to each pipeline a record of type:

`input { pipeline { address => myVirtualAddress2 } }`

In a separate file, I this not working...

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [August 6, 2020, 1:22pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/6 "2020-08-06T13:22:47Z")

</div>

The only thing I can think of off the top of my head is that each input can only be used once (as far as I know). So any specific port or _virtual address_ can only be used once, otherwise I would expect there to be resource conflicts between pipelines. I would expect to see that in Logstash logs though.

Could you show your file and folder structure and the `config/pipelines.yml` you use.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 3, 2020, 1:22pm UTC](https://discuss.elastic.co/t/problem-with-cloning-filebeat-logstash-elasticsearch-more-logstash-pipelines/244018/7 "2020-09-03T13:22:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.