# Why does Logstash not restart from JDBC I/O errors? **URL:** \ **Category:** Logstash\ **Created:** [January 7, 2018, 7:43pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422 "2018-01-07T19:43:17Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [January 7, 2018, 7:43pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/1 "2018-01-07T19:43:17Z")
For some reason lately I've been experiencing JDBC input I/O errors in Logstash's logs, but Logstash evidently doesn't do anything when it encounters these. It just hangs and the pipeline can't continue anymore. I have to manually check if it's still processing and when I see Elasticsearch isn't indexing anymore I go back to Logstash, see the I/O error log, restart it and it works for another few million docs or so. Couldn't Logstash see these errors and restart the connection automatically when it experiences them? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 8, 2018, 11:03am UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/2 "2018-01-08T11:03:44Z")
What version of the JDBC input are you using? `bin/logstash-plugin list --verbose | grep jdbc` or similar. This version `4.3.0` was refactored to open and close connections for each trip to the db (each scheduled run) ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [January 8, 2018, 2:16pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/3 "2018-01-08T14:16:20Z")
> logstash-filter-jdbc\_streaming (1.0.3) > logstash-input-jdbc (4.3.2) I could be wrong but I feel like this I/O error doesn't break the connection or something. I think I remember seeing it was still connected to the database I'm pulling from, but the logs were just stopped on that I/O error and nothing else was happening. I had to stop Logstash to kill the connection. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 8, 2018, 3:11pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/4 "2018-01-08T15:11:23Z")
Is there any other backtrace info in the logs? Is there any indication in the Postgres logs that it restarted a little bit before the log message. Is there a network proxy or load balancer between LS and the DB server? Some prior art: `Caused by: java.io.EOFException` [https://www.postgresql.org/message-id/001701c53a90$cf879cc0$6600a8c0@kermit](https://www.postgresql.org/message-id/001701c53a90%24cf879cc0%246600a8c0@kermit) [https://www.postgresql.org/message-id/42540D4D.2000504%40hogranch.com](https://www.postgresql.org/message-id/42540D4D.2000504%40hogranch.com) `Caused by: java.net.SocketException: Socket closed` > [https://confluence.atlassian.com/confkb/getting-a-psqlexception-error-an-io-error-occured-while-sending-to-the-backend-due-to-corrupt-postgres-database-162988249.html](https://confluence.atlassian.com/confkb/getting-a-psqlexception-error-an-io-error-occured-while-sending-to-the-backend-due-to-corrupt-postgres-database-162988249.html) -\> advice: check that the postgres db is not crashing (hw failure) You could try to set `jdbc_properties` via the `sequel_opts` setting... ```auto sequel_opts => { "jdbc_properties" => { "socketTimeout" => 20 #seconds "tcpKeepAlive" => true } } ``` See [https://jdbc.postgresql.org/documentation/94/connect.html](https://jdbc.postgresql.org/documentation/94/connect.html) ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [January 8, 2018, 6:51pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/5 "2018-01-08T18:51:14Z")
> Is there any other backtrace info in the logs? I've included our Postgres logs below. > Is there any indication in the Postgres logs that it restarted a little bit before the log message. I don't think so, but I'm not sure. The logs I've included are where the errors start. > Is there a network proxy or load balancer between LS and the DB server? No > 2018-01-08 16:17:30 UTC:removed[8341]:LOG: server process (PID 21535) was terminated by signal 11: Segmentation fault > > 2018-01-08 16:17:30 UTC:removed[8341]:LOG: terminating any other active server processes > > 2018-01-08 16:17:30 UTC:removed(48672):removed@removed:[10278]⚠ terminating connection because of crash of another server process > > 2018-01-08 16:17:30 UTC:removed(48672):removed@removed:[10278]:DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. > > 2018-01-08 16:17:30 UTC:removed(48672):removed@removed:[10278]:HINT: In a moment you should be able to reconnect to the database and repeat your command. repeat above logs a bunch of times > 2018-01-08 16:17:30 UTC:removed[8341]:LOG: archiver process (PID 26438) exited with exit code 1 > > 2018-01-08 16:17:30 UTC:removed(53072):removed@removed:[21450]⚠ terminating connection because of crash of another server process > > 2018-01-08 16:17:30 UTC:removed(53072):removed@removed:[21450]:DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. > > 2018-01-08 16:17:30 UTC:removed(53072):removed@removed:[21450]:HINT: In a moment you should be able to reconnect to the database and repeat your command. then repeat below log a bunch of times > 2018-01-08 16:17:30 UTC:removed(53166):removed@removed:[21537]:FATAL: the database system is in recovery mode Do you think that this log: > 2018-01-08 16:17:30 UTC:removed(53072):removed@removed:[21450]:DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. could be because I often run `sudo systemctl stop logstash` and maybe the Postgres database couldn't stop the large query very quickly so the processed was killed in a bad way (or something)? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 9, 2018, 9:33am UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/6 "2018-01-09T09:33:21Z")
Well it does look like your Postgres db is segfaulting. Sequel uses a connection pool under the hood so I guess the jdbc input's connection is a pool connection lease. I think it is these connections that are throwing the PGException indirectly. ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [January 9, 2018, 12:56pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/7 "2018-01-09T12:56:03Z")
I'm not familiar with the concept of segfaulting. Is that something caused by a bug in Postrgesql, or AWS RDS, or is it because of some misconfiguration on our part? Or it normal to see this if you overload the database somehow? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 9, 2018, 1:04pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/8 "2018-01-09T13:04:17Z")
> **[Segmentation fault](https://en.wikipedia.org/wiki/Segmentation_fault)** > > In computing, a segmentation fault (often shortened to segfault) or access violation is a fault, or failure condition, raised by hardware with memory protection, notifying an operating system (OS) the software has attempted to access a restricted area of memory (a memory access violation). On standard x86 computers, this is a form of general protection fault. The OS kernel will, in response, usually perform some corrective action, generally passing the fault on to the offending process by sending... An abnormal obscure bug of some kind in Postgres I think. **How often does it happen?** I am not a DBA type expert. Try to upgrade to a more recent version of PG and the driver too. Other than that, you will need to put your Sherlock Holmes hat on and search the internet. ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [January 9, 2018, 1:57pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/9 "2018-01-09T13:57:55Z")
okay thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 6, 2018, 1:58pm UTC](https://discuss.elastic.co/t/why-does-logstash-not-restart-from-jdbc-i-o-errors/114422/10 "2018-02-06T13:58:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.