Take events from one pipeline and forward to another pipeline : Pipeline to Pipeline Communication

[rkhapre (RK)]

Hi All

I have a scenario, wherein i need to read events from one data source and transfer the output of this pipeline to other pipeline as a sprintf like %{abc}, please let me know how to achieve this

Here is my example conf file

pipeline1.cfg
input{
file {
path => ".../flatfile.csv"
#sincedb_path => ".../sincedb.txt"
type => "csv"
start_position => "beginning"
}
}
filter {
csv {
columns => ["abc","def","ghi"]
separator => ","
}

}
output { elasticsearch { ... } }

This is second pipeline

pipeline2.cfg

input {
http_poller{
urls => {
http_request => {
method => get
url => "https://abcdefghjo.........."
headers => {
"authorization" => "%{abc}"
"Content-Type" => "application/xml; charset=UTF-8"
}
}
}

    }

}
filter { grok { ... } }
output { elasticsearch { ... } }

In the second pipeline, i am applying the authorization value %{abc}, which is coming from pipeline1, how to achieve this?

Thanks!

[Christian_Dahlqvist]

You can not feed incoming events into an input plugin (they would come through the pipeline input and go straight to the filters), so I don't think what you are trying to do is possible. For this to work you would probably need some kind of http_poller filter, but as far as I know no such filter exists.

[rkhapre (RK)]

Thanks for your reply

This is common scenario, i have lot of uses cases like this, wherein i need to talk with 2 pipelines. I thought this is possible with new 6.x version.

This is i am doing because my Authorization token expires every time. Do you have any solution wherein i can keep my authentication ALIVE?

Thanks

[Christian_Dahlqvist]

You can send data between pipelines, but not trigger input plugins based on the data.

[rkhapre (RK)]

okay not sure what to do, looks like this is not right tool to achieve this.
Anyways .Thanks for your response

[Christian_Dahlqvist]

One option might be to create a custom plugin, possibly based on the http_poller plugin, which seems close to fitting your requirements.

[rkhapre (RK)]

Yes, you are correct. I started working in same direction. But i am not expert of plugin development.
I am trying something with ruby filter script wherein i can execute http

Thanks!