Jdbc_input_plugin : extra count query with paging disabled

Elastic Stack Logstash
1

I am trying to setup a JDBC input plugin.
I have turned off paging :

jdbc_paging_enabled => false
jdbc_page_size => 0

Yet, in the "debug" logs I can see

SELECT count(*) "COUNT" FROM ( [my_query] ) "T1" FETCH NEXT 1 ROWS ONLY

This is a problem for me, since the query will never return the same thing twice.

Since Logstash does not work in cluster, we are trying to "split" the work at the one place all instances are accessing : the DB. We built a query that will return a different chunk everytime, and paging is handled in the DB size. Since each query will return a different result, you can understand why the extra "count" query is problematic

The goal is to indexing X millions records, splitting the load across logstash nodes, and don't index anything twice.

Is there a way to disable this extra "count" query?

2

I do not think so. The open issue is still open.

3

I found a workaround.

I used a “prepared statement” with no parameters.

Thanks for pointing out the issue on github.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.