Pipeline To Pipeline Communication inside Rancher

[mohsin106 (Mohsin)]

Hi, I need some guidance on setting up pipeline-to-pipeline communication inside of Rancher. What config files do I need at a minimum? I'm confused as to whether or not I need both logstash.yml and pipelines.yml configured or just one of them. I'm guessing I need both. Logstash.yml file should contain my xpack authentication parameters and my pipelines.yml should contain the multiple pipeline information each with its own set of input, filter, and output plugins?

[mohsin106 (Mohsin)]

I believe I figured it out. Logstash.yml is needed, and you can configure the path.queue setting and point it to the volume inside of the Logstash container that you configured as a persistent volume in Rancher. In my case I attached my persistent volume to the /usr/share/logstash/data directory. So I used the same directory to store queue data which you need when you want to implement pipeline-to-pipeline communication using the forked pattern.

path.queue: "/usr/share/logstash/data"

Then you also need pipelines.yml which will contain something similar to the below:

- pipeline.id: pipeline1
   queue.type: persisted
   config.string: |
     input { your input settings go here }
     filter { your filter settings go here }
     output { pipeline { send_to => "my-pl-1" } }     ********

In order to pass the event from the first pipeline "pipeline1" to a second pipeline "pipeline2" you need to point the output {} filter of the first pipeline to a "virtual address" (my-pl-1 in this ex.) that the second pipeline will use as it's input:

 - pipeline.id: pipeline2
   queue.type: persisted
   config.string: |
     input { pipeline { address => "my-pl-1" } }      ********
     filter { your filter settings go here }
     output { your output settings go here }

I configured logstash.yml and pipelines.yml in my config-map file and then mounted that config-map to the /usr/share/logstash/config directory inside of the Logstash container.