Issue with elastic search 5.0.0 - java.lang.ClassNotFoundException: org.apache.logging.log4j.core.async.DaemonThreadFactory

lao · October 29, 2016, 3:32pm

ERROR Could not reconfigure JMX java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/async/DaemonThreadFactory
at org.apache.logging.log4j.core.jmx.Server.createExecutor(Server.java:81)
at org.apache.logging.log4j.core.jmx.Server.(Server.java:66)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:541)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:603)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:226)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:49)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:105)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:72)
at org.elasticsearch.common.component.AbstractComponent.(AbstractComponent.java:37)
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:110)
at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:81)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:106)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:228)
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:69)
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:65)

pom.xml:

	<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch -->
	<dependency>
		<groupId>org.elasticsearch</groupId>
		<artifactId>elasticsearch</artifactId>
		<version>5.0.0</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.elasticsearch.client/transport -->
	<dependency>
		<groupId>org.elasticsearch.client</groupId>
		<artifactId>transport</artifactId>
		<version>5.0.0</version>
	</dependency>
	<dependency>
		<groupId>org.apache.logging.log4j</groupId>
		<artifactId>log4j-api</artifactId>
		<version>2.7</version>
	</dependency>
	<dependency>
		<groupId>org.apache.logging.log4j</groupId>
		<artifactId>log4j-core</artifactId>
		<version>2.7</version>
	</dependency>

jprante · October 29, 2016, 10:28pm

Have you tried the system property -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
as documented in Log4j – Log4j 2 Lock-free Asynchronous Loggers for Low-Latency Logging ?

jprante · October 29, 2016, 10:32pm

And you need to add a dependency to disruptor 3+, of course

<dependency>
    <groupId>com.lmax</groupId>
    <artifactId>disruptor</artifactId>
    <version>3.3.6</version>
</dependency>

fabiocatalao · October 31, 2016, 10:51am

I'm having the same problem with org/apache/logging/log4j/core/async/DaemonThreadFactory.

I've added the attached dependencies to my pom;
I've created a basic log42j.xml config file;
I've set the DLog4jContextSelector system property.

Is there anything that I'm forgetting to do?

<dependencies>
	<dependency>
		<groupId>org.elasticsearch.client</groupId>
		<artifactId>transport</artifactId>
		<version>5.0.0</version>
	</dependency>
	<dependency>
		<groupId>org.elasticsearch</groupId>
		<artifactId>elasticsearch</artifactId>
		<version>5.0.0</version>
	</dependency>
	<dependency>
		<groupId>org.elasticsearch.plugin</groupId>
		<artifactId>transport-netty4-client</artifactId>
		<version>5.0.0</version>
		<type>jar</type>
	</dependency>
	<dependency>
		<groupId>org.apache.logging.log4j</groupId>
		<artifactId>log4j-api</artifactId>
		<version>2.7</version>
	</dependency>
	<dependency>
		<groupId>org.apache.logging.log4j</groupId>
		<artifactId>log4j-core</artifactId>
		<version>2.7</version>
	</dependency>
	<dependency>
		<groupId>com.lmax</groupId>
		<artifactId>disruptor</artifactId>
		<version>3.3.6</version>
	</dependency>
</dependencies>

fabiocatalao · October 31, 2016, 12:26pm

In order to run my app, I've temporarily disabled JMX.
I've now the following arguments:

-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
-Dlog4j2.disable.jmx=true
-Dlog4j.configurationFile=log4j2.xml

msorah · November 1, 2016, 7:12pm

Change the log4j versions to 2.6.2. DaemonThreadFactory is not in version 2.7 of log4j but is in 2.6.2

escay · November 3, 2016, 10:29am

Using log4j 2.6.2 indeed seems to work, I used the following dependencies for running integration tests:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.dummy</groupId>
    <artifactId>test.es.integration.tests</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.8</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.test</groupId>
            <artifactId>framework</artifactId>
            <version>5.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.6.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.6.2</version>
        </dependency>
    </dependencies>
</project>

Using log4j 2.7 I can make it work by disabling the JMX as follows:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.dummy</groupId>
    <artifactId>test.es.integration.tests</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19.1</version>
                <configuration>
                    <systemPropertyVariables>
                        <!-- Disable JMX logging, because of Could not reconfigure JMX java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/async/DaemonThreadFactory -->
                        <log4j2.disable.jmx>true</log4j2.disable.jmx>
                    </systemPropertyVariables>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.8</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.test</groupId>
            <artifactId>framework</artifactId>
            <version>5.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.7</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.7</version>
        </dependency>
    </dependencies>
</project>

I also tried setting the

<Log4jContextSelector>org.apache.logging.log4j.core.async.AsyncLoggerContextSelector</Log4jContextSelector>

and adding the disruptor dependency. This only works when the Log4J jmx is disabled. Otherwise I still get the

ERROR Could not reconfigure JMX java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/async/DaemonThreadFactory

It also seems ES 5.0 is expecting Log4j version 2.6.2 and not yet 2.7.
See https://github.com/elastic/elasticsearch/pull/20805 with targets of ES 5.1 an ES 6.0.
So it is probably advised to use the 2.6.2 version and choose yourself if you want the low latency logging or not.

jasontedor · November 4, 2016, 12:22pm

The issue here is indeed that you have the wrong version of the Log4j dependency, it should be 2.6.2, not 2.7 for the 5.0.0 release. The other stuff about async and the LMAX disruptor is not the problem. This is the ideal option.

If for some reason this option is not available to you, you can also set the system property log4j2.is.webapp=true or log4j2.jmx.notify.async=false (either will be fine).

anand_rathi · January 12, 2017, 1:57am

After adding correct log4j configuration this error goes away

org.apache.logging.log4j
log4j-api
2.7

org.apache.logging.log4j
log4j-core
2.7