Hi,
I am using Logstash 8.10.0 with the HTTP input plugin and having 2 outputs, one using the HTTP output plugin and the other using the file output. The file rotation will occur every 1 hour, and my system-level ulimit is 1024. The issue I am facing is that the file descriptor (fd) process count slowly increases, and at one point, I encountered an issue with the -max-open-file limit. I have attached the log below. When I check the fd processes in the system using the command lsof -p , it shows more open HTTP client connections.
[02fcae19dd9ab97b6431010414614ae6dcab442c743fce4135ae08c42515f7df] Closing file /logger/IRELAND/LOGSTASH/OUTPUT_LOG_1/file-2024-06-28-01.log
[2024-06-28T02:07:04,851][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:06,634][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:09,681][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:17,127][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:22,525][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:25,923][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:30,364][WARN ][io.netty.channel.DefaultChannelPipeline][main][events-in-input-http] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.Net.accept(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:391) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:119) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:116) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:154) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:79) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) [?:?]
[2024-06-28T02:07:30,442][ERROR][logstash.javapipeline ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"/opt/product/IRELAND/logstash-8.11.1/queue/data/main/checkpoint.head.tmp (Too many open files)", :exception=>Java::JavaIo::FileNotFoundException, :backtrace=>["java.base/java.io.FileOutputStream.open0(Native Method)", "java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)", "java.base/java.io.FileOutputStream.(FileOutputStream.java:235)", "java.base/java.io.FileOutputStream.(FileOutputStream.java:184)", "org.logstash.ackedqueue.io.FileCheckpointIO.write(FileCheckpointIO.java:105)", "org.logstash.ackedqueue.Page.forceCheckpoint(Page.java:230)", "org.logstash.ackedqueue.Page.headPageCheckpoint(Page.java:195)", "org.logstash.ackedqueue.Page.checkpoint(Page.java:184)", "org.logstash.ackedqueue.Page.ack(Page.java:164)", "org.logstash.ackedqueue.Queue.ack(Queue.java:730)", "org.logstash.ackedqueue.Batch.close(Batch.java:60)", "org.logstash.ackedqueue.AckedBatch.close(AckedBatch.java:56)", "org.logstash.ackedqueue.AckedReadBatch.close(AckedReadBatch.java:92)", "org.logstash.execution.QueueReadClientBase.closeBatch(QueueReadClientBase.java:132)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:89)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base/java.lang.reflect.Method.invoke(Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:329)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:193)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:193)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:350)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"], :thread=>"#<Thread:0x6eaad682 /opt/product/IRELAND/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2024-06-28T02:07:30,485][WARN ][logstash.javapipeline ][main] Waiting for input plugin to close {:pipeline_id=>"main", :thread=>"#<Thread:0x6eaad682 /opt/product/IRELAND/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-06-28T02:07:37,029][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
please guide me for the same