Pipeline terminates in jdbc static filter plugin


(manjur) #1

I have used this config file to perform joins on two tables. I used jdbc static filter plug in .When i run logstash it terminates.

input {
}

filter {
jdbc_static {
loaders => [
{
id => "tbl_farmer"
query => "select farmerid from tbl_farmer"
local_table => "localfarmer"
},
{
id => "tbl_cultivation"
query => "select cultivationid,farmerid from tbl_cultivation"
local_table => "localcultivation"
}
]
local_db_objects => [
{
name => "localfarmer"
index_columns => ["farmerid"]
columns => [
["farmerid", "int"]
]
},
{
name => "localcultivation"
index_columns => ["cultivationid"]
columns => [
["cultivationid", "int"],
["farmerid", "int"]
]
}
]
local_lookups => [
{
query => "select localfarmer.farmerid as fname,localcultivation.cultivationid as culname from localfarmer inner join localcultivation on localfarmer.farmerid=localcultivation.farmerid"
target => "server"
}
]
add_field => { farmmnme => "%{[server][0][fname]}" }
add_field => { farmname => "%{[server][1][culname]}" }
jdbc_connection_string => "jdbc:sqlserver://L-MANJURG4067;databaseName=Cultivation;integratedSecurity=true;"
jdbc_driver_library => "C:\Users\manjur.gani\Desktop\ElasticSearch\sqljdbc_6.4\enu\mssql-jdbc-6.4.0.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"

}
}

output{
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => ["localhost:9200"]
index => "demopoc1"
}
}


(Christian Dahlqvist) #2

What does you input block look like?


(manjur) #3

i have given jdbc connection in input section.

input {
jdbc {
jdbc_driver_library => "C:\Users\manjur.gani\Desktop\ElasticSearch\sqljdbc_6.4\enu\mssql-jdbc-6.4.0.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://L-MANJURG4067;databaseName=Cultivation;integratedSecurity=true;"
}
}
}


(Christian Dahlqvist) #4

As you have not specified any schedule, I believe it will terminate once all the input data has been retrieved by the input plugin. What you are seeing is therefore expected as far as I can tell.


(manjur) #5

Thank you Christian for the reply. Can you please suggest how i should create config file structure . so that i will able to achieve the desired result. For example i have a table A ,Table B . i want to make joins in logstash config file. I can fetch table records using jdbc driver in the input sections. But how i will able to join the those two different data sets in filter sections.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.