APM Agent is not working on FIPS enabled java application

Dhanaraju_Vasireddy · January 21, 2021, 3:48pm

Hi
We are trying to using APM agent in an application which is FIPS complaint and using CCJ jar (security.provider.5=com.sun.net.ssl.internal.ssl.Provider CCJ)

Problem1: In the Java APM agent, below line keeps failing as "SSL SSLContext is not available"
SSLContext sslContext = SSLContext.getInstance("SSL");
without CCJ everything works but with CCJ, we only have TLS and no SSL.

Fix1: We tested the agent by changing to below code an it worked.
SSLContext sslContext = SSLContext.getInstance("TLS");

Problem2: Even after above fix, it complained about X509TrustManager with error " KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used"

Fix2: We changed createTrustAllTrustManager() to get trust manager array object using below code

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(
            TrustManagerFactory.getDefaultAlgorithm());
        trustManagerFactory.init((KeyStore) null);
        trustManagerFactory.getTrustManagers();

Now we are able to connect on FIPS configured application.

We are not sure if this is APM agent issue or we can easily resolve using some configuration without code changes, can someone please help.

Sylvain_Juge · January 26, 2021, 9:35am

Hi @Dhanaraju_Vasireddy, welcome to our forum !

That looks something that we should be able to support. The fact that you were able to make it work with a few changes is already very promising, thanks a lot for investigating this !

To be honest, I am not very familiar with FIPS settings, thus we may require some experimentation to ensure it's working properly and without any regression.

I'm not sure to understand the changes you did to the createTrustAllTrustManager() method, could you provide a full copy of the method body (or even better open a PR or provide a complete diff of your changes) ?

Eyal_Koren · January 26, 2021, 11:43am

Also, if you can provide the full errors and stack traces you got before each change, that may be useful as well.
Thanks!

Dhanaraju_Vasireddy · January 26, 2021, 3:30pm

Hi @Sylvain_Juge, @Eyal_Koren

Thank you for checking,

Below is the stack trace if I use APM agent without changes

2021-01-26 10:20:16,651 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation SSLContextInstrumentation: ((((name(equals(getDefault)) and isPublic()) and isStatic()) and hasParameter(ofSize(0))) and not(isAbstract())) matches public static synchronized javax.net.ssl.SSLContext javax.net.ssl.SSLContext.getDefault() throws java.security.NoSuchAlgorithmException
2021-01-26 10:20:16,764 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ssl.SslUtils - SSL SSLContext not available
java.security.NoSuchAlgorithmException: SSL SSLContext not available
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159) ~[?:1.8.0_162]
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.ssl.SslUtils.createSocketFactory(SslUtils.java:87) [?:?]
        at co.elastic.apm.agent.report.ssl.SslUtils.<clinit>(SslUtils.java:57) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.startRequestToUrl(ApmServerClient.java:140) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:278) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:69) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:47) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2021-01-26 10:20:16,771 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ssl.SslUtils - SSL SSLContext not available
java.security.NoSuchAlgorithmException: SSL SSLContext not available
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159) ~[?:1.8.0_162]
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.ssl.SslUtils.createSocketFactory(SslUtils.java:87) [?:?]
        at co.elastic.apm.agent.report.ssl.SslUtils.<clinit>(SslUtils.java:59) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.startRequestToUrl(ApmServerClient.java:140) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:278) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:69) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:47) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2021-01-26 10:20:16,773 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to https://XXXXX/
2021-01-26 10:20:16,774 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server https://XXXXXX/config/v1/agents
2021-01-26 10:20:16,817 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.report.ApmServerClient - Exception while interacting with APM Server, trying next one.
2021-01-26 10:20:16,817 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server https://XXXXXX/ is not available (java.lang.NullPointerException)
2021-01-26 10:20:16,819 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - null
2021-01-26 10:20:16,821 [elastic-apm-server-healthcheck] ERROR co.elastic.apm.agent.util.ExecutorUtils - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.HttpUtils.consumeAndClose(HttpUtils.java:68) ~[?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:283) ~[?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:69) ~[?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:47) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.NullPointerException
        at sun.net.www.protocol.https.HttpsClient.createSocket(HttpsClient.java:405) ~[?:1.8.0_162]
        at sun.net.NetworkClient.doConnect(NetworkClient.java:162) ~[?:1.8.0_162]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_162]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:1.8.0_162]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.ApmServerHealthChecker$1.withConnection(ApmServerHealthChecker.java:77) ~[?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker$1.withConnection(ApmServerHealthChecker.java:69) ~[?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:279) ~[?:?]
        ... 6 more
2021-01-26 10:20:16,823 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2021-01-26 10:20:17,072 [main] DEBUG co.elastic.apm.agent.impl.circuitbreaker.GCStressMonitor - Registering a heap memory pool (PS Eden Space) for stress monitoring
2021-01-26 10:20:17,072 [main] DEBUG co.elastic.apm.agent.impl.circuitbreaker.GCStressMonitor - Registering a heap memory pool (PS Old Gen) for stress monitoring
2021-01-26 10:20:17,074 [main] DEBUG co.elastic.apm.agent.impl.circuitbreaker.SystemCpuStressMonitor - Successfully obtained reference to the getSystemCpuLoad method of this JVM's OperatingSystemMXBean implementation

Also below is the modified code to get around the issue,

  @Nullable
    private static SSLSocketFactory createSocketFactory(TrustManager[] trustAllCerts) {
        try {
            try {
                SSLContext sslContext = SSLContext.getInstance("SSL");
                sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
                return sslContext.getSocketFactory();
            }
            catch (NoSuchAlgorithmException e) {
                logger.warn(e.getMessage(), e);
                logger.warn("Using TLS as SSL is not available");
                SSLContext sslContext = SSLContext.getInstance("TLS");
                sslContext.init(null, (trustAllCerts!=null)?getTrustManagers():null, new java.security.SecureRandom());
                return sslContext.getSocketFactory();
            }
        } catch (NoSuchAlgorithmException | KeyManagementException | KeyStoreException e) {
            logger.warn(e.getMessage(), e);
            return null;
        }
    }

    private static TrustManager[] getTrustManagers() throws NoSuchAlgorithmException, KeyStoreException {
        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(
            TrustManagerFactory.getDefaultAlgorithm());

        trustManagerFactory.init((KeyStore) null);
        return trustManagerFactory.getTrustManagers();
    }

    private static X509TrustManager createTrustAllTrustManager() {
        return new X509TrustManager() {
            @Override
            public void checkClientTrusted(X509Certificate[] chain, String authType) {
            }

            @Override
            public void checkServerTrusted(X509Certificate[] chain, String authType) {
            }

            @Override
            public X509Certificate[] getAcceptedIssuers() {
                return new X509Certificate[0];
            }
        };
    }

Eyal_Koren · January 26, 2021, 4:07pm

Thanks, that's very helpful!
Do you happen to have the error you got after applying only the first change? A similar log with error message and stack trace like you provided would be great.

Dhanaraju_Vasireddy · January 26, 2021, 4:58pm

Hi @Eyal_Koren
Below is the error log if I just read TLS context and leave everything as is.

2021-01-26 11:45:54,241 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorInvokeAnyAllInstrumentation: (((((((((hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor)))) and not(name(contains(jetty)))) and not(name(contains(tomcat)))) and not(name(contains(jboss)))) and not(name(contains(undertow)))) and not(name(contains(netty)))) and not(name(startsWith(com.hazelcast)))) and not((name(contains($Proxy)) or name(contains($$))))) and not(isInterface())) matches class java.util.concurrent.Executors$FinalizableDelegatedExecutorService
2021-01-26 11:45:54,245 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ssl.SslUtils - FIPS mode: only SunJSSE TrustManagers may be used
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used
        at sun.security.ssl.SSLContextImpl.chooseTrustManager(SSLContextImpl.java:115) ~[?:1.8.0_162]
        at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:78) ~[?:1.8.0_162]
        at javax.net.ssl.SSLContext.init(SSLContext.java:282) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.ssl.SslUtils.createSocketFactory(SslUtils.java:93) [?:?]
        at co.elastic.apm.agent.report.ssl.SslUtils.<clinit>(SslUtils.java:56) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.startRequestToUrl(ApmServerClient.java:140) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:278) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:69) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:47) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2021-01-26 11:45:54,255 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to https://XXXXX/
2021-01-26 11:45:54,255 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server https://XXXXXX/config/v1/agents
2021-01-26 11:45:54,270 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorRunnableInstrumentation: (((((((((hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor)))) and not(name(contains(jetty)))) and not(name(contains(tomcat)))) and not(name(contains(jboss)))) and not(name(contains(undertow)))) and not(name(contains(netty)))) and not(name(startsWith(com.hazelcast)))) and not((name(contains($Proxy)) or name(contains($$))))) and not(isInterface())) matches class java.util.concurrent.Executors$DelegatedExecutorService
2021-01-26 11:45:54,270 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorCallableInstrumentation: (((((((((hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor)))) and not(name(contains(jetty)))) and not(name(contains(tomcat)))) and not(name(contains(jboss)))) and not(name(contains(undertow)))) and not(name(contains(netty)))) and not(name(startsWith(com.hazelcast)))) and not((name(contains($Proxy)) or name(contains($$))))) and not(isInterface())) matches class java.util.concurrent.Executors$DelegatedExecutorService
2021-01-26 11:45:54,272 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation ExecutorInvokeAnyAllInstrumentation: (((((((((hasSuperType(erasure(name(equals(java.util.concurrent.Executor)))) and not(name(equals(org.apache.felix.resolver.ResolverImpl$DumbExecutor)))) and not(name(contains(jetty)))) and not(name(contains(tomcat)))) and not(name(contains(jboss)))) and not(name(contains(undertow)))) and not(name(contains(netty)))) and not(name(startsWith(com.hazelcast)))) and not((name(contains($Proxy)) or name(contains($$))))) and not(isInterface())) matches class java.util.concurrent.Executors$DelegatedExecutorService
2021-01-26 11:45:54,280 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server https://XXXXX/ is not available (java.lang.NullPointerException)

 private static SSLSocketFactory createSocketFactory(TrustManager[] trustAllCerts) {
        try {
            try {
                SSLContext sslContext = SSLContext.getInstance("SSL");
                sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
                return sslContext.getSocketFactory();
            }
            catch (NoSuchAlgorithmException e) {
                logger.warn(e.getMessage(), e);
                logger.warn("Using TLS as SSL is not available");
                SSLContext sslContext = SSLContext.getInstance("TLS");
                sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
                return sslContext.getSocketFactory();
            }
        } catch (NoSuchAlgorithmException | KeyManagementException e) {
            logger.warn(e.getMessage(), e);
            return null;
        }
    }

We are using CCJ jar file and below is our java.security file

#
# List of providers and their preference orders (see above):
#

security.provider.1=com.safelogic.cryptocomply.jcajce.provider.CryptoComplyFipsProvider
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider CCJ
security.provider.6=com.sun.crypto.provider.SunJCE
security.provider.7=sun.security.jgss.SunProvider
security.provider.8=com.sun.security.sasl.Provider
security.provider.9=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.10=sun.security.smartcardio.SunPCSC

securerandom.source=file:/dev/random

securerandom.strongAlgorithms=NativePRNGBlocking:SUN

login.configuration.provider=sun.security.provider.ConfigFile

policy.provider=sun.security.provider.PolicyFile

policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy

policy.expandProperties=true

policy.allowSystemProperty=true

policy.ignoreIdentityScope=false

keystore.type=BCFKS

keystore.type.compat=true

package.access=sun.,\
               com.sun.xml.internal.,\
               com.sun.imageio.,\
               com.sun.istack.internal.,\
               com.sun.jmx.,\
               com.sun.media.sound.,\
               com.sun.naming.internal.,\
               com.sun.proxy.,\
               com.sun.corba.se.,\
               com.sun.org.apache.bcel.internal.,\
               com.sun.org.apache.regexp.internal.,\
               com.sun.org.apache.xerces.internal.,\
               com.sun.org.apache.xpath.internal.,\
               com.sun.org.apache.xalan.internal.extensions.,\
               com.sun.org.apache.xalan.internal.lib.,\
               com.sun.org.apache.xalan.internal.res.,\
               com.sun.org.apache.xalan.internal.templates.,\
               com.sun.org.apache.xalan.internal.utils.,\
               com.sun.org.apache.xalan.internal.xslt.,\
               com.sun.org.apache.xalan.internal.xsltc.cmdline.,\
               com.sun.org.apache.xalan.internal.xsltc.compiler.,\
               com.sun.org.apache.xalan.internal.xsltc.trax.,\
               com.sun.org.apache.xalan.internal.xsltc.util.,\
               com.sun.org.apache.xml.internal.res.,\
               com.sun.org.apache.xml.internal.resolver.helpers.,\
               com.sun.org.apache.xml.internal.resolver.readers.,\
               com.sun.org.apache.xml.internal.security.,\
               com.sun.org.apache.xml.internal.serializer.utils.,\
               com.sun.org.apache.xml.internal.utils.,\
               com.sun.org.glassfish.,\
               com.oracle.xmlns.internal.,\
               com.oracle.webservices.internal.,\
               oracle.jrockit.jfr.,\
               org.jcp.xml.dsig.internal.,\
               jdk.internal.,\
               jdk.nashorn.internal.,\
               jdk.nashorn.tools.,\
               jdk.xml.internal.,\
               com.sun.activation.registries.,\
               com.sun.browser.,\
               com.sun.glass.,\
               com.sun.javafx.,\
               com.sun.media.,\
               com.sun.openpisces.,\
               com.sun.prism.,\
               com.sun.scenario.,\
               com.sun.t2k.,\
               com.sun.pisces.,\
               com.sun.webkit.,\
               jdk.management.resource.internal.


package.definition=sun.,\
                   com.sun.xml.internal.,\
                   com.sun.imageio.,\
                   com.sun.istack.internal.,\
                   com.sun.jmx.,\
                   com.sun.media.sound.,\
                   com.sun.naming.internal.,\
                   com.sun.proxy.,\
                   com.sun.corba.se.,\
                   com.sun.org.apache.bcel.internal.,\
                   com.sun.org.apache.regexp.internal.,\
                   com.sun.org.apache.xerces.internal.,\
                   com.sun.org.apache.xpath.internal.,\
                   com.sun.org.apache.xalan.internal.extensions.,\
                   com.sun.org.apache.xalan.internal.lib.,\
                   com.sun.org.apache.xalan.internal.res.,\
                   com.sun.org.apache.xalan.internal.templates.,\
                   com.sun.org.apache.xalan.internal.utils.,\
                   com.sun.org.apache.xalan.internal.xslt.,\
                   com.sun.org.apache.xalan.internal.xsltc.cmdline.,\
                   com.sun.org.apache.xalan.internal.xsltc.compiler.,\
                   com.sun.org.apache.xalan.internal.xsltc.trax.,\
                   com.sun.org.apache.xalan.internal.xsltc.util.,\
                   com.sun.org.apache.xml.internal.res.,\
                   com.sun.org.apache.xml.internal.resolver.helpers.,\
                   com.sun.org.apache.xml.internal.resolver.readers.,\
                   com.sun.org.apache.xml.internal.security.,\
                   com.sun.org.apache.xml.internal.serializer.utils.,\
                   com.sun.org.apache.xml.internal.utils.,\
                   com.sun.org.glassfish.,\
                   com.oracle.xmlns.internal.,\
                   com.oracle.webservices.internal.,\
                   oracle.jrockit.jfr.,\
                   org.jcp.xml.dsig.internal.,\
                   jdk.internal.,\
                   jdk.nashorn.internal.,\
                   jdk.nashorn.tools.,\
                   jdk.xml.internal.,\
                   com.sun.activation.registries.,\
                   com.sun.browser.,\
                   com.sun.glass.,\
                   com.sun.javafx.,\
                   com.sun.media.,\
                   com.sun.openpisces.,\
                   com.sun.prism.,\
                   com.sun.scenario.,\
                   com.sun.t2k.,\
                   com.sun.pisces.,\
                   com.sun.webkit.,\
                   jdk.management.resource.internal.

security.overridePropertiesFile=true

ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=PKIX

networkaddress.cache.negative.ttl=10

krb5.kdc.bad.policy = tryLast

jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224


jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, DES40_CBC, RC4_40

jdk.tls.legacyAlgorithms= \
        K_NULL, C_NULL, M_NULL, \
        DH_anon, ECDH_anon, \
        RC4_128, RC4_40, DES_CBC, DES40_CBC, \
        3DES_EDE_CBC

jdk.xml.dsig.secureValidationPolicy=\
    disallowAlg http://www.w3.org/TR/1999/REC-xslt-19991116,\
    disallowAlg http://www.w3.org/2001/04/xmldsig-more#rsa-md5,\
    disallowAlg http://www.w3.org/2001/04/xmldsig-more#hmac-md5,\
    disallowAlg http://www.w3.org/2001/04/xmldsig-more#md5,\
    maxTransforms 5,\
    maxReferences 30,\
    disallowReferenceUriSchemes file http https,\
    minKeySize RSA 1024,\
    minKeySize DSA 1024,\
    noDuplicateIds,\
    noRetrievalMethodLoops

Eyal_Koren · January 27, 2021, 1:56pm

@Dhanaraju_Vasireddy thanks for reporting and providing such a useful information!

I think the first fix you applied is the one that is actually required. Your second fix would only be required if you set the verify_server_cert config option to false, which I assume is invalid in your case.
So what I did is just quietly catch the Exception thrown when trying to create a trust-all socket factory (and warn the user not to try and set verify_server_cert=false).

Please try this snapshot and see if it fixes the problem.

Thanks again!

Dhanaraju_Vasireddy · January 27, 2021, 4:23pm

Thank you @Eyal_Koren

I am able to test with your patch from my local machine and it is sending the data to my APM server. I see below warning but I think you meant to ignore this during FIPS mode, please confirm.

2021-01-27 10:22:23,909 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ssl.SslUtils - SSL is not supported, trying to use TLS instead.
2021-01-27 10:22:23,913 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ssl.SslUtils - FIPS mode: only SunJSSE TrustManagers may be used
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used
        at sun.security.ssl.SSLContextImpl.chooseTrustManager(SSLContextImpl.java:115) ~[?:1.8.0_162]
        at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:78) ~[?:1.8.0_162]
        at javax.net.ssl.SSLContext.init(SSLContext.java:282) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.ssl.SslUtils.createSocketFactory(SslUtils.java:107) [?:?]
        at co.elastic.apm.agent.report.ssl.SslUtils.<clinit>(SslUtils.java:64) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.startRequestToUrl(ApmServerClient.java:142) [?:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:269) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:69) [?:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:47) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2021-01-27 10:22:23,918 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to https://esapm-poc.saviyntcloud.com/

However when I deploy this jar from one of our AWS containers, getting below error and not publishing events to APM server.

"failed to validate metadata: error validating JSON: I[#] S[#] doesn't validate with \"metadata#\"\n  I[#/cloud/instance/name] S[#/properties/cloud/properties/instance/properties/name/type] expected string, but got null".

In the metadata we have name as null "instance":{"id":"i-XXXXXXXX","name":null}
It was working fine when I had verify cert as false and my other part of fix.

Eyal_Koren · January 28, 2021, 7:18am

You are right, it is just a leftover and should have no effect on you. I indeed decided to ignore an error with the creation of the trust-all socket factory as long as the user is not actively disabling server cert verification, which is only when this factory is used. I will remove this warning.

This is definitely a problem we need to fix! The reason you didn't see it before is that it is a brand new feature that was just merged to the Java agent and not released yet, but is included in the snapshot you tried. This feature is discovering cloud metadata and decorates APM events with it. As far as I am aware, the cloud.instance.name can be null, so I will look into why this happens.
Two questions that should assist:

What APM Server version are you using?
If you set log_level to DEBUG, you should see a line in the agent logs around startup that starts with: Cloud metadata discovered:. If not too difficult, please try to see if you get that and it includes the metadata info you are expecting.

Thanks for your awesome feedback!

Eyal_Koren · January 28, 2021, 10:49am

@Dhanaraju_Vasireddy you may be using APM server 7.9.x, where this validation was more restrictive that it should be. If that's the case, please try upgrading your APM server to 7.10 and try this snapshot with it.
Let us know if this resolves everything or not.

If you don't want to upgrade the APM server, you can continue running with your fixes and upgrade both agent (when 1.21.0 is released) and server when you are ready.

Dhanaraju_Vasireddy · January 28, 2021, 3:44pm

@Eyal_Koren with the APM server 7.10.2 (before I was using 7.9.2) which is sitting behind load balancer and with TLS SelfSignedCert disabled, we are getting below error with latest patch.

2021-01-28 15:39:11,946 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed), response code is -1
2021-01-28 15:39:11,946 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed)
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1551) ~[?:1.8.0_162]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:95) ~[?:1.8.0_162]
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_162]
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_162]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_162]
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) ~[?:1.8.0_162]
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[?:1.8.0_162]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:154) [?:?]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.endRequest(IntakeV2ReportingEventHandler.java:167) [?:?]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:83) [?:?]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:71) [?:?]
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:42) [?:?]
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [?:?]
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1921) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1885) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1830) ~[?:1.8.0_162]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128) ~[?:1.8.0_162]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_162]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_162]
        at java.io.PrintStream.flush(PrintStream.java:338) ~[?:1.8.0_162]
        at sun.net.www.http.ChunkedOutputStream.flush(ChunkedOutputStream.java:172) ~[?:1.8.0_162]
        at sun.net.www.http.ChunkedOutputStream.flush(ChunkedOutputStream.java:297) ~[?:1.8.0_162]
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[?:1.8.0_162]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3565) ~[?:1.8.0_162]
        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:149) [?:?]
        ... 7 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_162]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_162]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_162]
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[?:1.8.0_162]
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886) ~[?:1.8.0_162]
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857) ~[?:1.8.0_162]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_162]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_162]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_162]
        at java.io.PrintStream.flush(PrintStream.java:338) ~[?:1.8.0_162]
        at sun.net.www.http.ChunkedOutputStream.flush(ChunkedOutputStream.java:172) ~[?:1.8.0_162]
        at sun.net.www.http.ChunkedOutputStream.flush(ChunkedOutputStream.java:297) ~[?:1.8.0_162]
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[?:1.8.0_162]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_162]
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3565) ~[?:1.8.0_162]
        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241) ~[?:1.8.0_162]
        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:149) ~[?:?]
        ... 7 more
2021-01-28 15:39:11,948 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null

Eyal_Koren · January 31, 2021, 9:40am

@Dhanaraju_Vasireddy sorry about that
It is indeed something with your SSL/TLS settings.

Did you see this issue in the troubleshooting section? If you enabled client certificate authentication this may happen.

In any case, make sure to read through this SSL/TLS guide and hopefully you can figure this out. If not, please describe exactly your certificate settings both in APM Server and in the Java agent so we can think what went wrong.

Eyal_Koren · February 2, 2021, 3:38pm

@Dhanaraju_Vasireddy I would really wish to get you through all these hurdles and make sure you are successful with our solutions and are able to enjoy our latest features.

Please describe your entire SSL/TLS-related configurations (like - which certs are you using? where did you add them to? did you disable server certs authentication? etc.), so we can get you working with 7.10.

If you are still unable or prefer not to properly upgrade to 7.10, you can use this snapshot, which includes the FIPS support PR as well as a new workaround for 7.9.x users with our cloud metadata feature.
If fact, if you still have a 7.9 server, it would be great if you would test this snapshot anyway, so that we can verify no other user will have to go through this inconvenience.

Dhanaraju_Vasireddy · February 2, 2021, 4:01pm

@Eyal_Koren thanks for checking. I am in process of setting new setup with 7.10.2 properly from scratch, will keep you posted.

Eyal_Koren · February 3, 2021, 5:25am

Looking forward to hear back.
If you still have APM Server 7.9.x available on the AWS environment and you can test the latest agent snapshot with it, that would be super useful.
Thanks

Dhanaraju_Vasireddy · February 3, 2021, 6:37am

Thanks @Eyal_Koren We are able to test with your patch and 7.10.2 without issues.

Eyal_Koren · February 3, 2021, 6:56am

Awesome, these are great news!
Then the FIPS support will be released soon with 1.21.0, and it should work properly both with APM Server 7.10.x (as you verified) as well as 7.9.x (which I understand you cannot verify as you no longer have it available).

Dhanaraju_Vasireddy · February 3, 2021, 7:03am

Sure @Eyal_Koren will use 1.21.0 on 7.10.2 when it is available. Thanks again for quick help and support

Sylvain_Juge · February 3, 2021, 8:42am

If you haven't already, make sure to subscribe to the releases on Github repository GitHub - elastic/apm-agent-java: Elastic APM Java Agent that would ensure that you'll get notified once the new release is available.

system · February 24, 2021, 4:43am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.