Logstash - Level deserialization failed - SOLVED

Elastic Stack Logstash
1

Hello,

I am facing a new issue with my Logstash configuration.
It follows the previous subject I opened : Log4j SocketAppender. Logstash : tcp input --> OK, log4j input --> KO - SOLVED - #20 by wiibaa

I am receiving the logs and they are properly de-serialized as I can see them via Kibana.
But I am receiving a lot of errors in the stderr.log file (always the same) :

log4j:WARN Level deserialization failed, reverting to default.
java.lang.NoSuchMethodException: com.eibus.util.logger.Severity.toLevel(int)
    at java.lang.Class.getDeclaredMethod(Class.java:2130)
    at org.apache.log4j.spi.LoggingEvent.readLevel(LoggingEvent.java:436)
    at org.apache.log4j.spi.LoggingEvent.readObject(LoggingEvent.java:463)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
    at org.jruby.ext.jruby.JRubyObjectInputStream.readObject(JRubyObjectInputStream.java:56)
    at org.jruby.ext.jruby.JRubyObjectInputStream$INVOKER$i$0$0$readObject.call(JRubyObjectInputStream$INVOKER$i$0$0$readObject.gen)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:56)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
    at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
    at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
    at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
    . . .
    at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
    at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
    at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
    at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
    at org.jruby.runtime.Block.call(Block.java:101)
    at org.jruby.RubyProc.call(RubyProc.java:300)
    at org.jruby.RubyProc.call(RubyProc.java:230)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
    at java.lang.Thread.run(Thread.java:745)

Where I am surprised is that I have included the following .jar files in the Log4j.rb file :

require "E:\App\ELK\CordysLogger\managementlib.jar"
require "E:\App\ELK\CordysLogger\log4j-1.2.15.jar"

In the managementlib.jar, I can see :

package com.eibus.util.logger;

import java.util.Locale;
import org.apache.log4j.Level;

public class Severity extends Level
{

public static final Severity FATAL = new Severity(50000, "FATAL", 0);
public static final Severity ERROR = new Severity(40000, "ERROR", 3);
public static final Severity WARN = new Severity(30000, "WARN", 4);
public static final Severity INFO = new Severity(20000, "INFO", 6);
public static final Severity DEBUG = new Severity(10000, "DEBUG", 7);

private Severity(int severity, String severityStr, int syslogEquivalent)
{
    super(severity, severityStr, syslogEquivalent);
}

public static final Severity toSeverity(String level)
{
    String logLevel = level.toUpperCase(Locale.ENGLISH);
    Severity severity = FATAL;
    if("DEBUG".equals(logLevel))
    {
        severity = DEBUG;
    } else
    if("INFO".equals(logLevel))
    {
        severity = INFO;
    } else
    if("WARN".equals(logLevel))
    {
        severity = WARN;
    } else
    if("ERROR".equals(logLevel))
    {
        severity = ERROR;
    }
    return severity;
}

}

So I can see that the Severity class extends the Level one.
By checking in the log4j-1.2.15.jar (that is also included in the Log4j.rb file), I found the Level class and the following toLevel method :

public static Level toLevel(int val)
{
    return toLevel(val, DEBUG);
}

Can someone help me with this Java issue ?
Thanks in advance !

2

Hello, I have an explanation but not a solution

Quoting from log4j source code, see http://grepcode.com/file/repo1.maven.org/maven2/log4j/log4j/1.2.14/org/apache/log4j/spi/LoggingEvent.java#359

// Note that we use Class.getDeclaredMethod instead of
// Class.getMethod. This assumes that the Level subclass
// implements the toLevel(int) method which is a
// requirement. Actually, it does not make sense for Level
// subclasses NOT to implement this method. Also note that
// only Level can be subclassed and not Priority.

So the library you are using does not meet the expectation of Log4J, the hackish solution would be to create a custom jar where you would write the missing method in the Severity class or maybe you can report an issue to the vendor.

As you mentioned, the input still work because log4j catch the exception and use a default value but the repeated exception raising would slow down the log ingestion, so this would be nice to fix.

3

Hi wiibaa,

Thanks for your reply.
I am bit confused. Java programming's years are far now !
To which vendor are you refering to ? Log4j's one ?

Anyway, if I want to use the hackish way to solve the issue, can you please provide more details ?
Cause at the moment, this is what I understand (correct me if I am wrong) :

  • The Logstash Log4j's input is using the getDeclaredMethod to find the toLevel(int) method.
  • The commented notes you provided mentionned that the toLevel(int) method must be implemented in the Level subclass.
  • By adding the managementlib.jar as required lib in the log4j.rb file, the Severity class that extend the Level one is available.
  • By adding the log4j-1.2.15.jar as required lib in the log4j.rb file, the Level class is available.
  • As the toLevel(int) method is implemented in the Level class, why am I still receiving the error message ?
    In other term, the getDeclaredMethod method should returned the toLevel(int) method, isn't it ?
4

It is easy to summarize the issue without going too deep in Java Reflection API,.

Log4j "contract" expects that the class Severity (from managementlib.jar) must define its OWN method toLevel(int) ! so inheriting it from class Level (from log4j.jar) is not enough to fullfill this contract.
In practice, it use Class.getDeclaredMethod to scan the methods exactly declared inside class Severity (without looking up into super-classes) searching for a toLevel(int) method and does not find one.

Either the author/vendor of managementlib.jar can fix this for you or you should try to hack it yourself from the sources if available or with a decompiler. If you can share this jar file, I could have a quick look, to validate it is feasible

5

Hi wiibaa,

Please find the managementlib.jar file available at this link.
Thanks in advance !

6

Here is a very quick tentative to a modified jar, https://www.dropbox.com/s/cftkqdcye1lohjr/managementlib.jar?dl=0
It is compiled using java 1.8, normally the default when running logstash.

What I did in a temporary directory

  • Put the original managementlib.jar

  • Put log4j1.2.15.jar

  • Create a folder structure com/eibus/util/logger

  • Create a file Severity.java under the logger folder created before with content:.

    package com.eibus.util.logger;

    import java.util.Locale;
    import org.apache.log4j.Level;

    public class Severity extends Level {

      public static final Severity FATAL = new Severity(50000, "FATAL", 0);
  public static final Severity ERROR = new Severity(40000, "ERROR", 3);
  public static final Severity WARN = new Severity(30000, "WARN", 4);
  public static final Severity INFO = new Severity(20000, "INFO", 6);
  public static final Severity DEBUG = new Severity(10000, "DEBUG", 7);

  private Severity(int severity, String severityStr, int syslogEquivalent) {
    super(severity, severityStr, syslogEquivalent);
  }

  public static final Severity toSeverity(String level) {
    String logLevel = level.toUpperCase(Locale.ENGLISH);
    Severity severity = FATAL;
    if("DEBUG".equals(logLevel))
    {
        severity = DEBUG;
    } else
    if("INFO".equals(logLevel))
    {
        severity = INFO;
    } else
    if("WARN".equals(logLevel))
    {
        severity = WARN;
    } else
    if("ERROR".equals(logLevel))
    {
        severity = ERROR;
    }
    return severity;
  }

  //Added method
  public static Severity toLevel(int val) {
    switch (val) {
      case 0:
        return FATAL;
      case 3:
        return ERROR;
      case 4:
        return WARN;
      case 6:
        return INFO;
      case 7:
        return DEBUG;
      default:
        return DEBUG;
    }
  }

    }

  • Compile the class and update the jar archive (run from the temporary dir, not inside the com/... dir)

    javac -classpath log4j-1.2.15.jar com/eibus/util/logger/Severity.java
    jar uvf managementlib.jar com/eibus/util/logger/Severity.class

  • That's it

1 Like
7

IT WORKS !