Hi Folks,
for Elasticsearch logging we use log4j2, which is configured to send logs to log server.
We had issues, that we fixed implementing .java.policy file (as decribed in this topic: Log4j framework can no longer reconnect because the security manager on logstash restart )
Now, as we upgraded Elasticsearch from 8.12.3 to 8.18.3, we again have problems with sending logs.
[2025-08-25T06:34:05,876][ERROR][StatusLogger ] [client-logs-iop1] [pdb-iop1-ces-1] Unable to write to stream TCP:logs-1.my-domain.local:12201 for appender GELFSocket
org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:logs-1.my-domain.local:12201 for connection logs-1.my-domain.local/10.41.51.195:12201
at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:244) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:190) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.writeByteArrayToManager(AbstractOutputStreamAppender.java:224) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.SocketAppender.directEncodeEvent(SocketAppender.java:513) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:208) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:199) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.dispatch(AsyncAppenderEventDispatcher.java:130) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.dispatchAll(AsyncAppenderEventDispatcher.java:92) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.appender.AsyncAppenderEventDispatcher.run(AsyncAppenderEventDispatcher.java:74) ~[elasticsearch-log4j-8.18.3.jar:?]
Caused by: java.net.SocketException: Broken pipe
at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?]
at sun.nio.ch.NioSocketImpl.tryWrite(NioSocketImpl.java:393) ~[?:?]
at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:409) ~[?:?]
at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:439) ~[?:?]
at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:813) ~[?:?]
at java.net.Socket$SocketOutputStream.implWrite(Socket.java:1098) ~[?:?]
at java.net.Socket$SocketOutputStream.write(Socket.java:1085) ~[?:?]
at org.apache.logging.log4j.core.net.TcpSocketManager.writeAndFlush(TcpSocketManager.java:253) ~[elasticsearch-log4j-8.18.3.jar:?]
at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:219) ~[elasticsearch-log4j-8.18.3.jar:?]
... 12 more
As I found out, ES 8.18 is on Java 24, and on this Java version "System properties related to SecurityManager are now ignored, and the conf/security/java.policy file has been removed." - so .java.policy file doesn't help anymore.
This is part of log4j2.properties file:
# ==== GELF Socket Appender (TCP) - Tuned and wrapped by Async ====
appender.socket.name = GELFSocket
appender.socket.type = Socket
appender.socket.protocol = tcp
appender.socket.host = logs-iop1.io-ancotel.local
appender.socket.port = 12201
# Retry and buffering configurations
appender.socket.immediateFail = false
appender.socket.reconnectDelayMillis = 5000
appender.socket.bufferSize = 16384
appender.socket.connectTimeoutMillis = 5000
appender.socket.retryIntervalMillis = 5000