I am trying to get spark logs to logstash with custom log4j2 properties. My log4j2 properties file
#Define the log4j configuration for local application
#log4j.rootLogger=INFO, server
#We will use socket appender
log4j.appender.server=org.apache.log4j.net.SocketAppender
#Port where socket server will be listening for the log events
#log4j.appender.server.Port=5044
log4j.appender.server.encoding=UTF-8
#Defining the layout
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.layout.conversionPattern=${spark.yarn.app.container.log.dir} [%d] %p %m (%c)%n
#Host name or IP address of socket server
log4j.appender.server.RemoteHost=192.168.85.176
#Define any connection delay before attempting to reconnect
log4j.appender.server.ReconnectionDelay=10000
Logstash configuration file
input {
tcp {
port => 5044
codec => plain {
charset => "UTF-8"
}
}
}
filter{}
output {
stdout { codec => rubydebug }
}
I am getting from logstash stdout
"port" => 37438,
"@timestamp" => 2021-06-15T06:44:57.495Z,
"host" => "slave1",
"message" => "threadNameq\u0000~\u0000\u0001L\u0000\rthrowableInfot\u0000+Lorg/apache/log4j/spi/ThrowableInformation;xp\u0000\u0000\u0000\u0000\u0001z\u000EjzIt\u00006org.apache.spark.network.client.TransportClientFactorypppt\u00006Creating new connection to slave2/192.168.85.177:51795t\u0000\u0016netty-rpc-connection-0pw\u0004\u0000\u0000'\u0010pxy",
"@version" => "1"
As in here there are some decoding issue . Appreciate your help .