Take events from one pipeline and forward to another pipeline : Pipeline to Pipeline Communication

Hi All

I have a scenario, wherein i need to read events from one data source and transfer the output of this pipeline to other pipeline as a sprintf like %{abc}, please let me know how to achieve this

Here is my example conf file

pipeline1.cfg
input{
file {
path => ".../flatfile.csv"
#sincedb_path => ".../sincedb.txt"
type => "csv"
start_position => "beginning"
}
}
filter {
csv {
columns => ["abc","def","ghi"]
separator => ","
}

}
output { elasticsearch { ... } }

This is second pipeline

pipeline2.cfg

input {
http_poller{
urls => {
http_request => {
method => get
url => "https://abcdefghjo.........."
headers => {
"authorization" => "%{abc}"
"Content-Type" => "application/xml; charset=UTF-8"
}
}
}

    }

}
filter { grok { ... } }
output { elasticsearch { ... } }

In the second pipeline, i am applying the authorization value %{abc}, which is coming from pipeline1, how to achieve this?

Thanks!

You can not feed incoming events into an input plugin (they would come through the pipeline input and go straight to the filters), so I don't think what you are trying to do is possible. For this to work you would probably need some kind of http_poller filter, but as far as I know no such filter exists.

Thanks for your reply

This is common scenario, i have lot of uses cases like this, wherein i need to talk with 2 pipelines. I thought this is possible with new 6.x version.

This is i am doing because my Authorization token expires every time. Do you have any solution wherein i can keep my authentication ALIVE?

Thanks

You can send data between pipelines, but not trigger input plugins based on the data.

okay not sure what to do, looks like this is not right tool to achieve this.
Anyways .Thanks for your response

One option might be to create a custom plugin, possibly based on the http_poller plugin, which seems close to fitting your requirements.

Yes, you are correct. I started working in same direction. But i am not expert of plugin development.
I am trying something with ruby filter script wherein i can execute http

Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.