Can I pass data between 2 input plugin of logstash


(Patrick) #1

input {

kafka {

// Pass a value from kafka to jdbc input plugin
}

jdbc {

}

}


(Christian Dahlqvist) #2

No, that is not possible. What is it you are looking to achieve?


(Patrick) #3

I want fetch a message (and index value (primary_key of a table )) from Kafka Queue and pass this jdbc input so that i can query it and send the data to Elasticsearch


(Christian Dahlqvist) #4

How many records/documents would each query return? If you are simply looking up a value based on the key coming from Kafka, you may be able to use the jdbc_streaming plugin for this.


(Patrick) #5

Only One key at a time,
You mean I can use Kafka in the Input plugin and then use the Jdbc stream plugin to take input value from kafka and then run a query in the filter section and then in ouput section use elasticsearch right ?

input{
kakfa {}
}

filter {
jdbcstreaming
}

output {
elasticsearch {}
}

Is this right ? Can you share example if you have any ?


(Christian Dahlqvist) #6

Something like that. Do not have any example though.


(Patrick) #7

But this is possible if implemented ?


(Walker) #8

@Christian_Dahlqvist, is it not possible to configure a TCP output for the first 'stage' and then a TCP input for the second 'stage'?


(Christian Dahlqvist) #9

@patrick007 I think it should be possible, but I do not know anything about your data or queries.

@wwalker I am not sure I understand what you are suggesting. Could you please add some more details on why TCP inputs would be useful here?


(Patrick) #11

@Christian_Dahlqvist
Thanks for you input :+1:
Used a wrong emoticon first :sweat_smile::sweat_smile:


(Kilynn W) #12

Input from Kafka topic, enrich, output to another Kafka topic.... rinse lather repeat


(Patrick) #13

@sm00thindian Did not follow, Can you explain more please


(system) #14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.