How to chain multiple input in Logstash

My use case is something like below

First Input(Mysql JDBC)
I'm selecting some data from MySQL like

select pk_id, score, other_id from <some_table>

Second Chained Input (Elasticsearch)
Then I want to pass the data pk_id list got from the first input to elasticsearch to get other information from it

Not exactly like this but just to fetch other data from es
some_index/_search
{
    "query": {
         "terms": {
                "values": []
         }
    }
}

Then combine the response from elasticsearch to the records obtained from the first input with the corresponding pk_id

<pk_id>, score, other_id, <other_info_from_elasticsearch>

Output Elasticsearch
Output the above records to the elasticsearch document

Can I do this using some plugin in logstash? or I have to do that using custom ruby script in logstash

You cannot reference events from one input in another, but you can use an elasticsearch filter to do this.

1 Like

Thanks Badger, it solved the issue but I have one more question.
It is calling elasticsearch for single events. So let's say I'm getting 1000 records at once from JDBC input then filter will process them one by one and will make 1000 calls to elasticasearch and then it is going to output plugin.
But somehow the output plugin uses the bulk API for it can I do something so that filter plugin will also call it in bulk to avoid huge number of HTTP calls
@Badger

I do not think logstash can do that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.