JDBC Input Plugin Count I/O Error

Elastic Stack Logstash
1

I have a rather large query that will be picking up several million records from a Postgres DB and indexing into Elasticsearch. I have always used the jdbc input plugin for taking data from a RDBMS and putting it into Elasticsearch. This process now, however, is running a count with my query embedded as a sub query. This ends up running for so long that it never completes and causes an I/O error to be thrown.

I don't really understand the necessity of having this within the jdbc input plugin, but has anyone experienced this and found a work around for it?

I tried manually running this count query on my own outside of logstash and I stopped it after it had attempted to run for over 30 minutes.

2

Looking at the ruby code for the jdbc input filter, there is a file 'checked_count_logger.rb' which has the following:

return unless @in_debug
check_count_query(query) if @needs_check

and within the initialize method in that same file:

@in_debug = @logger.debug?

Is there a config setting or jdbc input setting to force this to be FALSE so that count doesn't get run?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.