Jdbc input plugin doesn't work with prepared statements enabled and large amount of data is proceeding

Before I'll create an issue on Github I'd like to be sure that I'm dealing here with a bug and not with some configuration problem.

I've a task to import data from Oracle database to Elasticsearch. Of course I'm using JDBC input plugin for it.
Due to performance reasons I need to enable prepared statements for a plugin.
(It will reduce read operations on DB and will proper index usage)
My configuration looks as follows:
input {
jdbc {
jdbc_fetch_size => 999
schedule => "* * * * *"
use_prepared_statements => true
prepared_statement_name => "foo"
prepared_statement_bind_values => [":sql_last_value"]
statement => " SELECT
......
FROM table_name tbl
JOIN ......
JOIN ...
LEFT JOIN ......
WHERE tbl.id > ?
"
use_column_value => true
tracking_column => "id"
}
}

But here I hit a problem. After activating it:

  • no events are transmitted in logstash
  • no new documents in ELK are created
  • CPU usage and memory consumptions is 100%.
  • after some time logstash scrash with following error:
    java.lang.BootstrapMethodError: call site initialization exception

Few important remarks:

  • it doesn't matter if I change jdbc_fetch_size to smaller or larger value (it only affects how fast memory will be consumed)
  • on smaller amount of data everything works fine - documents in ELK indexes are created but with slight delay which doesn't occur when prepared statement are disabled.
    - when prepared statement are disabled everything works fine even with large data and without any delay:

Am I dealing here with a bug?

1 Like

I also think this error is not real root cause but only result of out of memory error or something similar.

Did anybody use jdbc plugin, prepared statements and large data volume together?

use sql query instead. I have complex query and large record returning

statement_filepath => "/etc/logstash/conf.d/sql/myown.sql"

But why it should help? Are prepared statements handled differently if statement is given within the file?

you can try to test if your sql has a problem or something else.

also when you use this it is more managable. your config file looks neat and clean.
your choice

It changes nothing, I can test my SQL also when it is within 'statement' property as well.
And I do not need to have it neat and clean for now when whole solution is not working for me...

alright good luck then. you tone is very harsh. people here are to help you on their own will.

Small note: I've tested it on two versions 7.8.0 and 7.9.0 - results on both is the same - not working.

Has anyone some other ideas how to help here?

*bump (just to not be forgotten and closed)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.