Jdbc_streaming filter plugin returns an empty set using parameters option

Elastic Stack Logstash
1

HELLO,

I'm using the jdbc input plugin to get data from "table1":

statement => "SELECT * FROM table1 where id=1"
Result is : id:1 and id_subscriber:1

Then I'm using the jdbc_streaming filter plugin to get more data from "table2" using the "id_subscriber" field value from the previous statement, so I'm using the following statement :

statement => "SELECT * FROM table2 where id_subscriber = :idsub"
parameters => { "idsub" => "%{id_subscriber}"}
target => "arrayOfResults" #arrayOfResults is an empty array

While the folowing statement returns the four records I'm looking for :

statement => "SELECT * FROM table2 where id_subscriber = 1"
target => "arrayOfResults" # I get the right result

Can you tell me what Im I doing wrong ?

Thank you.

This is a part of the code :

input {
  jdbc {
    jdbc_connection_string => 'jdbc:mysql://MYSERVER
    ETC...
    statement => "SELECT * FROM table1 where id=1"
 }
  }
filter{
 jdbc_streaming  
 {  
    jdbc_connection_string => 'jdbc:mysql://MYSERVER2
    ETC...
    # NON WORKING STATEMENT
    statement => "SELECT * FROM table2 where id_subscriber = :idsub"
    # WORKING STATEMENT -->   statement => "SELECT * FROM table2 where id_subscriber = 1"
    parameters => { "idsub" => "%{id_subscriber}"}
    target => "arrayOfResults"
  }
  json_encode {
    source => "arrayOfResults"
    target => "bar"
  }
  mutate {
    add_field => { "arrayOfResults" => "%{bar}" }
  } 
}
output {
  stdout { codec => json_lines }
  elasticsearch {
  'hosts' => 'localhost:9200'
  }
}
2

Hello

My bad, I did not understand how the "parameters" option works.
The right answer if someone came across this issue is simply :
parameters => { "idsub" => "id_subscriber"}

The right sided part of parameters was referring to the field name and not it's value.

3

Yes. You only need to use the %{id_subscriber} type syntax when you need to create a string made of two fields or add a prefix/suffix.

The real reason why %{id_subscriber} did not work for you is that %{id_subscriber} will convert the numeric id_subscriber value into a string via interpolation and then the statement fails as 1 is not equal to "1".

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.