I have a rather large query that will be picking up several million records from a Postgres DB and indexing into Elasticsearch. I have always used the jdbc input plugin for taking data from a RDBMS and putting it into Elasticsearch. This process now, however, is running a
count with my query embedded as a sub query. This ends up running for so long that it never completes and causes an I/O error to be thrown.
I don't really understand the necessity of having this within the jdbc input plugin, but has anyone experienced this and found a work around for it?
I tried manually running this count query on my own outside of logstash and I stopped it after it had attempted to run for over 30 minutes.