Logstash: Calling WebService or a program from filters

Elastic Stack Logstash
1

I am working on logstash to extract data from mysql,
as existing filters/plugins lack in doing required operations/transformations
I think I need to create a program of my own to do this.

My question is, Is it possible to call a java program/function or a webservice from filter?

Thanks for your help.

2

I'm using the JDBC input with a mysql connector. The plugin creates fields for each column I specify in the select statement. Would this not work for you?
As to your question though I don't know if you can call such from a filter but I've used the exec input to make a curl call.

3

There's also https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html and https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html

4

Thanks @CraigFoote and @warkolm for giving your time.

@warkolm I've gone through both the URL's you've mentioned and was really helpful.

I would also like to know if there is another way of calling external program/function (could be in any language) so that I can see which is efficient and will use.

@CraigFoote Can you please share a sample for using exec input for making url call and getting data from that?

Thanks once again for your help.

5

Thanks @warkolm for giving your time.

@warkolm I've gone through both the URL's you've mentioned and was really helpful.

I would also like to know if there is another way of calling external program/function (could be in any language) so that I can see which is efficient and will use.

Thanks once again for your help.

6

In my case we had a REST service that returned the JSON I needed so my logstash config was quite simple:

input{
    exec{
        command => "curl 'http://restservice.com'"
        codec => json
        interval => 60
    }
}
filter{
}
output{
    elasticsearch{
        ...
    }
}

But I imagine you could call a script or even a java program that sysouts json. Haven't tried it though.