Logstash input plugin log4j not working in unix

Elastic Stack Logstash
1

I am facing Serialization errors with logstash 5.2.1 while trying to use log4j and socket appender.

I have a custom log4j layouts/appenders which writes json objects as string while emmiting json string with socket appender logstash gives me Serialization errors.

Logstash config file

`input {
log4j {
port => 9099
}
}

filter{
json {
source => "message"
}
}`

2017-04-28 15:02:01,117 Ruby-0-Thread-16: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:494 ERROR Serialization of LogEvent failed. java.io.NotSerializableException: org.jruby.RubyHash$RubyHashEntry
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.apache.logging.log4j.core.layout.SerializedLayout.toByteArray(SerializedLayout.java:62)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.writeByteArrayToManager(AbstractOutputStreamAppender.java:127)
    at org.apache.logging.log4j.core.appender.SocketAppender.directEncodeEvent(SocketAppender.java:252)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:113)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:104)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:155)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:128)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:119)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:375)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:359)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:349)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
    at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2005)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1876)
    at org.apache.logging.log4j.spi.AbstractLogger.debug(AbstractLogger.java:437)
    at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:421)
    at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:288)
    at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:27)
    at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:50)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)
    at rubyjit.LogStash::Logging::Logger$$debug_a481093baf0bb2972c3adea031c44dd9fd49b4cd1028566121.__file__(/usr/share/logstash/logstash-core/lib/logstash/logging/logger.rb:44)
    at rubyjit.LogStash::Logging::Logger$$debug_a481093baf0bb2972c3adea031c44dd9fd49b4cd1028566121.__file__(/usr/share/logstash/logstash-core/lib/logstash/logging/logger.rb)
    at org.jruby.ast.executable.AbstractScript.__file__(AbstractScript.java:42)
    at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
    at org.jruby.ast.AndNode.interpret(AndNode.java:97)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
    at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
    at org.jruby.ast.IfNode.interpret(IfNode.java:116)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
    at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
    at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
    at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
    at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
    at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
    at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
    at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
    at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
    at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
    at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
    at org.jruby.runtime.Block.call(Block.java:101)
    at org.jruby.RubyProc.call(RubyProc.java:300)
    at org.jruby.RubyProc.call(RubyProc.java:230)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
    at java.lang.Thread.run(Thread.java:745)

Thanks for the help

2

You shouldn't use the log4j input. It's brittle and will be removed in future Logstash releases. There's been one or two recent blog posts about this.

3

Thanks Magnus, how about TCP plugin? same issue when I use TCP input plugin.

4

I find it hard to believe that you get exactly the same error when using a tcp input. Please show the resulting log.

5

You are right it is not the exact same error, I am trying to use socketAppender with custom jsonlayout here..

sample json

Sample Json

The below will be produced by jsonlayout as a single line though. also this json is been added to log4j MDC with 'message' as key.

{
"componentAddress":"xxx",
"componentLifecycle":"development",
"countryScope":"US","
eventTimestamp":"1493828490665",
"serviceScope":"ABC",
"eventLevel":"INFO",
"eventDescription":"some message",
"user":"Afroz"
}

Error log:

    12:01:12.127 [[main]>worker3] DEBUG logstash.pipeline - output received {"event"=>{"@timestamp"=>2017-05-03T16:01:10.884Z, "port"=>62213, "@version"=>"1", "host"=>"127.0.0.1", "message"=>"threadNameq\\u0000~\\u0000\\u0001L\\u0000\\rthrowableInfot\\u0000+Lorg/apache/log4j/spi/ThrowableInformation;xp\\u0000\\u0000\\u0000\\u0000\\u0001[\\xCF\\v8\\xB6t\\u0000\\u0004rootpsr\\u0000\\u0013java.util.Hashtable\\u0013\\xBB\\u000F%!J\\xE4\\xB8\\u0003\\u0000\\u0002F\\u0000", "tags"=>["_jsonparsefailure"]}}
    {"@timestamp":"2017-05-03T16:01:10.869Z","port":62213,"@version":"1","host":"127.0.0.1","message":"\\xAC\\xED\\u0000\\u0005sr\\u0000!org.apache.log4j.spi.LoggingEvent\\xF3\\xF2\\xB9#t\\v\\xB5?\\u0003\\u0000","tags":["_jsonparsefailure"]}{"@timestamp":"2017-05-03T16:01:10.884Z","port":62213,"@version":"1","host":"127.0.0.1","message":"Z\u0000\u0015mdcCopyLookupRequiredZ\u0000\u0011ndcLookupRequiredJ\u0000\ttimeStampL\u0000\fcategoryNamet\u0000\u0012Ljava/lang/String;L\u0000\flocationInfot\u0000#Lorg/apache/log4j/spi/LocationInfo;L\u0000\u0007mdcCopyt\u0000\u0015Ljava/util/Hashtable;L\u0000\u0003ndcq\u0000~\u0000\u0001L\u0000\u000FrenderedMessageq\u0000~\u0000\u0001L\u0000","tags":["_jsonparsefailure"]}{"@timestamp":"2017-05-03T16:01:10.884Z","port":62213,"@version":"1","host":"127.0.0.1","message":"threadNameq\\u0000~\\u0000\\u0001L\\u0000\\rthrowableInfot\\u0000+Lorg/apache/log4j/spi/ThrowableInformation;xp\\u0000\\u0000\\u0000\\u0000\\u0001[\\xCF\\v8\\xB6t\\u0000\\u0004rootpsr\\u0000\\u0013java.util.Hashtable\\u0013\\xBB\\u000F%!J\\xE4\\xB8\\u0003\\u0000\\u0002F\\u0000","tags":["_jsonparsefailure"]}12:01:12.396 [pool-2-thread-1] DEBUG logstash.instrument.collector - Collector: Sending snapshot to observers {:created_at=>2017-05-03 12:01:12 -0400}
6

SocketAppender sends serialized Java objects. You can't use a tcp input to deserialize them. You need to use another kind of appender that takes the event object, passes it to a layout, and sends the result over a socket connection.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.