Log4j framework can no longer reconnect because the security manager on logstash restart

[ansk98 (Anton Skripin)]

Hi all,

For the logging of the Elasticsearch processes, the log4j2 was configured to log server, slowlog indexing, and slowlog search information to logstash by adding a socket appender that sends relevant logs to the running logstash service.

The problem occurs if a logstash experiences an outage or must be restarted. In such a case, the following error can be seen in the logs of Elasticsearch:

ERROR An exception occurred processing Appender index_search_slowlog_rolling java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:4560" "connect,resolve")

It looks like whenever logstash processes are restarted, the log4j framework can no longer reconnect because of the security manager that is active in the Elasticsearch container by default.

Is there any way to bypass this issue? Currently, we are forced to restart the Elasticsearch container so that the logs are visible in the ELK again.

Here is the respective configuration for log4j that is used by Elasticsearch nodes:

status = error

######## Server JSON ############################
appender.rolling.type = Console
appender.rolling.name = rolling
appender.rolling.layout.type = ESJsonLayout
appender.rolling.layout.type_name = server

######## Socket Appender #######################
appender.socket.type = Socket
appender.socket.name = socket
appender.socket.host = localhost
appender.socket.port = 4560
appender.socket.reconnectionDelayMillis = 10000
appender.socket.connectTimeoutMillis = 5000
appender.socket.immediateFail = false
appender.socket.layout.type = ESJsonLayout
appender.socket.layout.type_name = server

################################################

################################################

rootLogger.level = info
rootLogger.appenderRef.rolling.ref = rolling
rootLogger.appenderRef.socket.ref = socket

######## Deprecation JSON #######################
appender.deprecation_rolling.type = Socket
appender.deprecation_rolling.host = localhost
appender.deprecation_rolling.port = 4560
appender.deprecation_rolling.name = deprecation_rolling
appender.deprecation_rolling.layout.type = ESJsonLayout
appender.deprecation_rolling.layout.type_name = deprecation.elasticsearch
appender.deprecation_rolling.layout.esmessagefields=x-opaque-id,key,category,elasticsearch.elastic_product_origin
appender.deprecation_rolling.filter.rate_limit.type = RateLimitingFilter

appender.header_warning.type = HeaderWarningAppender
appender.header_warning.name = header_warning
#################################################

#################################################
logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = WARN
logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
logger.deprecation.appenderRef.header_warning.ref = header_warning
logger.deprecation.additivity = false

######## Search slowlog JSON ####################
appender.index_search_slowlog_rolling.type = Socket
appender.index_search_slowlog_rolling.host = localhost
appender.index_search_slowlog_rolling.port = 4560
appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
appender.index_search_slowlog_rolling.layout.type = ESJsonLayout
appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog
appender.index_search_slowlog_rolling.layout.esmessagefields=message,took,took_millis,total_hits,types,stats,search_type,total_shards,source,id

#################################################

#################################################
logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = trace
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
logger.index_search_slowlog_rolling.additivity = false

######## Indexing slowlog JSON ##################
appender.index_indexing_slowlog_rolling.type = Socket
appender.index_indexing_slowlog_rolling.host = localhost
appender.index_indexing_slowlog_rolling.port = 4560
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
appender.index_indexing_slowlog_rolling.layout.type = ESJsonLayout
appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog
appender.index_indexing_slowlog_rolling.layout.esmessagefields=message,took,took_millis,doc_type,id,routing,source

Thanks in advance.

[ansk98 (Anton Skripin)]

Answering the question myself...

The following file must be mounted to the following directory inside the container: /usr/share/elasticsearch/.java.policy

The content of the file is:

grant {
    permission java.net.SocketPermission "localhost:4560", "connect,resolve";
};