Logstash heap out of memory

Hi ,

I am using logstash-7.7.0 to ingest my data from mongodb to elasticsearch using JDBC input plugin. I have around 17 configuration files which polling data from mongodb and the input section of all configuration files look similar like below.

I have placed below drivers in /usr/share/logstash/logstash-core/lib/jars path.

  1. mongo-java-driver-3.12.0.jar
  2. mongojdbc1.7.jar

input {
jdbc {
jdbc_driver_library =>""
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_connection_string => "jdbc:mongodb://user:password@mongoip1:32000,mongoip2:32001,mongoip3:32002/DB?authSource=admin&replicaSet=replica1&ssl=true&sslInvalidHostNameAllowed=true"
jdbc_user => ""
jdbc_validate_connection => true
jdbc_page_size => 200000
codec => "json"
schedule => "* * * * * UTC"
statement => "db.collection.find({ TimeStamp: { $gte: (:sql_last_value)}})"
last_run_metadata_path => "/usr/share/logstash/ddpa/.logstash_jdbc_last_run"
   }
}

My logstash server capacity is like below.

4 core , 28gb RAM.

Once I start logstash service it will run fine for 4 or 5 hours without any error and then I will get an heap out of memory error. Below is the exact error that I get.

Jun 29 13:39:03 nodemaster-0 logstash[1628]: Exception in thread "CleanCursors-2037-thread-1" java.lang.OutOfMemoryError: Java heap space
Jun 29 13:39:03 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2237-thread-1"
Jun 29 13:39:30 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2045-thread-1"
Jun 29 13:39:30 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2736-thread-1"
Jun 29 13:39:33 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2467-thread-1"

I have around 3 environments and in all the 3 its the same issue and I don't really understand why logstash gets restarted by itself after running for certain time in some environments.

I have seen many errors with logstash it never ran for more than 2 days without any error. I suspect that it is due to JDBC plugin which open the connection to mongodb and leaves it like that without closing. Sometimes I get below errors as well.

  1. Too many open files.
  2. Timeout connecting to server.(mongodb)

below are error logs for both.

Jun 29 06:26:00 nodemaster-0 logstash[35474]: [2020-06-29T06:26:00,737][INFO ][org.mongodb.driver.cluster][timeseries_devicedetails][a79106d0945bc8271cc32ebcde380ee32d81011e7c8d70a2719a326d0478f75c] Adding discovered server mongoip2:32001 to client view of cluster
Jun 29 06:26:00 nodemaster-0 logstash[35474]: [2020-06-29T06:26:00,901][INFO ][org.mongodb.driver.cluster][timeseries_devicedetails][a79106d0945bc8271cc32ebcde380ee32d81011e7c8d70a2719a326d0478f75c] Exception in monitor thread while connecting to server mongoip1:32000
Jun 29 06:26:00 nodemaster-0 logstash[35474]: com.mongodb.MongoSocketOpenException: Exception opening socket
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) [mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: Caused by: java.net.SocketException: Too many open files
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.createImpl(Socket.java:478) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.getImpl(Socket.java:538) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.setTcpNoDelay(Socket.java:998) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at sun.security.ssl.BaseSSLSocketImpl.setTcpNoDelay(BaseSSLSocketImpl.java:355) ~[?:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:38) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011... 3 more
Jun 29 06:26:09 nodemaster-0 logstash[35474]: [2020-06-29T06:26:09,142][ERROR][logstash.inputs.jdbc     ][timeseries][33bdfd6812e780ab76aa7944adbcc739f1059b8d98874fecd274026167c61eef] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLException: Timeout connecting the server."}
Jun 29 06:26:09 nodemaster-0 logstash[35474]:   15770     Java::JavaSql::SQLException: Timeout connecting the server.

I am struggling with this error for more than 2 months and I tried all the solutions that are available in elastic forums and stack overflow.

Help is really appreciated.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.