JDBC Input Plugin Count I/O Error

I have a rather large query that will be picking up several million records from a Postgres DB and indexing into Elasticsearch. I have always used the jdbc input plugin for taking data from a RDBMS and putting it into Elasticsearch. This process now, however, is running a count with my query embedded as a sub query. This ends up running for so long that it never completes and causes an I/O error to be thrown.

I don't really understand the necessity of having this within the jdbc input plugin, but has anyone experienced this and found a work around for it?

I tried manually running this count query on my own outside of logstash and I stopped it after it had attempted to run for over 30 minutes.

Looking at the ruby code for the jdbc input filter, there is a file 'checked_count_logger.rb' which has the following:

return unless @in_debug
check_count_query(query) if @needs_check

and within the initialize method in that same file:

@in_debug = @logger.debug?

Is there a config setting or jdbc input setting to force this to be FALSE so that count doesn't get run?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.