# How to connect multiple pipelines within the same Logstash instance running inside a docker container **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [May 31, 2021, 3:55am UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446 "2021-05-31T03:55:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Rahul\_Dey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_dey/32/89444_2.png) [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Post date:** [May 31, 2021, 3:55am UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/1 "2021-05-31T03:55:18Z")
Currently we have a data stream in elasticsearch, lets call it app\_stream in which we are ingesting log messages from an application XMLTransformer through FileBeats(port=5044) installed on the server where the application is running. We have a new scenario where now we are receiving log messages on the same FileBeats port 5044 from two different applications running on the same server. Let's call another application Model Server. Now log messages from both the XMLTransformer and Model Server are going into app\_stream. According to the requirement, we are suppose to create a new data stream ai\_model\_server which will only have log messages from Model Server whereas app\_stream will only have log\_messages from XML transformer. Also, we want to create two separate logstash pipeline for XMLTransformer and Model Server which will ingest the log messages from the same Filebeats port 5044 and we do not want to make any changes on the filebeats. So, we are trying to implement the **[Pipeline-to-Pipeline Communication](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#pipeline-to-pipeline)** to route the messages from app\_stream.conf pipeline to another ai\_model\_server.conf pipeline. But the logstash instance is running inside a docker container. How to get the Virtual Address which is available in docker container and use it in the logstash pipeline. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [May 31, 2021, 4:11am UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/2 "2021-05-31T04:11:33Z")
The virtual address is user defined. It's just a way to identify what pipeline to send the data to. See the example here [Pipeline-to-Pipeline Communication | Logstash Reference [7.13] | Elastic](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#distributor-pattern). ---
**Author:** ![Rahul\_Dey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_dey/32/89444_2.png) [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Post date:** [June 1, 2021, 7:39am UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/4 "2021-06-01T07:39:57Z")
@legoguy1000 So If I configure logstash like the following, will it work perfectly for my use case ??? I will be giving "modelserverlogs" literally as an virtual address in configuration as you mentioned that it can be user-defined (my confusion was whether I could give anything in place of the virtual address in the config or do I need to get the virtual IP addresses available in logstash docker container and use them in the config ) # ====================== app\_stream.conf input { beats { port =\> 5044 type =\> beats } } filter { #XML transformer filter statements add\_tag =\> ["xml\_transformer"] } output { if "xml\_transformer" in [tags] { elasticsearch { hosts =\> 0.0.0.0:0000 index =\> "app\_stream" } } else { pipeline { send\_to =\> modelserverlogs } } } # ====================== ai\_model\_server.conf input { pipeline { address =\> modelserverlogs } } filter { #modelserver log filter statements } output { elasticsearch { hosts =\> 0.0.0.0:0000 index =\> "ai\_model\_server" } } # ================= pipelines.yml - pipeline.id: app\_stream path.config: "${LOGSTASH\_BASE}/pipeline/app\_stream.conf" queue.type: persisted - pipeline.id: ai\_model\_server path.config: "${LOGSTASH\_BASE}/pipeline/ai\_model\_server.conf" queue.type: persisted ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [June 1, 2021, 1:23pm UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/5 "2021-06-01T13:23:15Z")
Next time you post code, use the code formatter so its easier to read. But looking at this, I believe it is correct and should work. ---
**Author:** ![Rahul\_Dey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_dey/32/89444_2.png) [@Rahul\_Dey](https://discuss.elastic.co/u/Rahul_Dey)\ **Post date:** [June 2, 2021, 7:46am UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/6 "2021-06-02T07:46:08Z")
@legoguy1000 It is working as expected. Thank you very much for your help. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [June 2, 2021, 12:25pm UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/7 "2021-06-02T12:25:21Z")
No problem, can you please mark the solution? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 30, 2021, 12:25pm UTC](https://discuss.elastic.co/t/how-to-connect-multiple-pipelines-within-the-same-logstash-instance-running-inside-a-docker-container/274446/8 "2021-06-30T12:25:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.