Hi ,
I am using logstash-7.7.0 to ingest my data from mongodb to elasticsearch using JDBC input plugin. I have around 17 configuration files which polling data from mongodb and the input section of all configuration files look similar like below.
I have placed below drivers in /usr/share/logstash/logstash-core/lib/jars path.
- mongo-java-driver-3.12.0.jar
- mongojdbc1.7.jar
input {
jdbc {
jdbc_driver_library =>""
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_connection_string => "jdbc:mongodb://user:password@mongoip1:32000,mongoip2:32001,mongoip3:32002/DB?authSource=admin&replicaSet=replica1&ssl=true&sslInvalidHostNameAllowed=true"
jdbc_user => ""
jdbc_validate_connection => true
jdbc_page_size => 200000
codec => "json"
schedule => "* * * * * UTC"
statement => "db.collection.find({ TimeStamp: { $gte: (:sql_last_value)}})"
last_run_metadata_path => "/usr/share/logstash/ddpa/.logstash_jdbc_last_run"
}
}
My logstash server capacity is like below.
4 core , 28gb RAM.
Once I start logstash service it will run fine for 4 or 5 hours without any error and then I will get an heap out of memory error. Below is the exact error that I get.
Jun 29 13:39:03 nodemaster-0 logstash[1628]: Exception in thread "CleanCursors-2037-thread-1" java.lang.OutOfMemoryError: Java heap space
Jun 29 13:39:03 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2237-thread-1"
Jun 29 13:39:30 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2045-thread-1"
Jun 29 13:39:30 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2736-thread-1"
Jun 29 13:39:33 nodemaster-0 logstash[1628]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "CleanCursors-2467-thread-1"
I have around 3 environments and in all the 3 its the same issue and I don't really understand why logstash gets restarted by itself after running for certain time in some environments.
I have seen many errors with logstash it never ran for more than 2 days without any error. I suspect that it is due to JDBC plugin which open the connection to mongodb and leaves it like that without closing. Sometimes I get below errors as well.
- Too many open files.
- Timeout connecting to server.(mongodb)
below are error logs for both.
Jun 29 06:26:00 nodemaster-0 logstash[35474]: [2020-06-29T06:26:00,737][INFO ][org.mongodb.driver.cluster][timeseries_devicedetails][a79106d0945bc8271cc32ebcde380ee32d81011e7c8d70a2719a326d0478f75c] Adding discovered server mongoip2:32001 to client view of cluster
Jun 29 06:26:00 nodemaster-0 logstash[35474]: [2020-06-29T06:26:00,901][INFO ][org.mongodb.driver.cluster][timeseries_devicedetails][a79106d0945bc8271cc32ebcde380ee32d81011e7c8d70a2719a326d0478f75c] Exception in monitor thread while connecting to server mongoip1:32000
Jun 29 06:26:00 nodemaster-0 logstash[35474]: com.mongodb.MongoSocketOpenException: Exception opening socket
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) [mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: Caused by: java.net.SocketException: Too many open files
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.createImpl(Socket.java:478) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.getImpl(Socket.java:538) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at java.net.Socket.setTcpNoDelay(Socket.java:998) ~[?:1.8.0_252]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at sun.security.ssl.BaseSSLSocketImpl.setTcpNoDelay(BaseSSLSocketImpl.java:355) ~[?:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:38) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongo-java-driver-3.12.0.jar:?]
Jun 29 06:26:00 nodemaster-0 logstash[35474]: #011... 3 more
Jun 29 06:26:09 nodemaster-0 logstash[35474]: [2020-06-29T06:26:09,142][ERROR][logstash.inputs.jdbc ][timeseries][33bdfd6812e780ab76aa7944adbcc739f1059b8d98874fecd274026167c61eef] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLException: Timeout connecting the server."}
Jun 29 06:26:09 nodemaster-0 logstash[35474]: 15770 Java::JavaSql::SQLException: Timeout connecting the server.
I am struggling with this error for more than 2 months and I tried all the solutions that are available in elastic forums and stack overflow.
Help is really appreciated.