Java APM Agent connecting to SSL APM Server

[Trinity_Rolling]

Kibana version: 7.15.1

Elasticsearch version: 7.15.1

APM Server version: 7.15.1

APM Agent language and version: Java 1.26.0

Browser version: Chrome 94.0.4606.81

Is there anything special in your setup?
Windows based, Windows Certificate Authority, SSL Certificates, Single node

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
APM Agent not communicating with APM Server.

Provide logs and/or server output (if relevant):

1. Where to find logs from APM Agent on APM Agent host if APM isn't communicating with the APM Server?

Documentation for securing APM Agent is vague. (SSL/TLS communication with APM Server | APM Java Agent Reference [1.x] | Elastic) Am I modifying my application's keystore in order to report back to the APM Server? This seems very intrusive and could potentially be a breaking change for the application especially an application that already uses SSL.

[riferrei (Ricardo Ferreira)]

Could you share a stacktrace from the JVM agent to better understand what is happening during the communication? If possible, please share as well your apm-server.yml configuration file.

Also, if you suspect that the problem is related to certificate validation, you can temporarily disable this behavior by using the following property in the JVM agent:

-Delastic.apm.verify_server_cert=false

It might be helpful to isolate the issue and see if cert mgmt is indeed what is causing the problem.

— @riferrei

[Trinity_Rolling]

I will start with adding the verify_server_cert=false to make sure there is not a certificate issue.

Many of the examples and tutorials show the APM Server as localhost. Is an APM Server required on each application host? I currently only have the APM Agent using the following settings with TomCat that runs as a Windows Service pointing to a remote APM Server.

-javaagent:E:\tomcat\elastic-apm\elastic-apm-agent-1.26.0.jar \
-Delastic.apm.service_name=myapp
-Delastic.apm.server_urls=https://elasticstack.mydomaincom:8200 \
-Delastic.apm.secret_token= AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2Vu... \
-Delastic.apm.environment=ist \
-Delastic.apm.application_packages=com.mydomain.myapp \

apm-server.yml

apm-server:
  host: "0.0.0.0:8200"
  auth:
    secret_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzU0N...
  #---------------------------- APM Server - Secure Communication with Agents ----------------------------
  ssl:
    enabled: true
    certificate: 'D:\Elasticsearch\config\elastic-server.cer'
    key: 'D:\Elasticsearch\config\elastic-server.key'
    supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]
  apm-server.auth.secret_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3R...
 #---------------------------- APM Server - Agent Configuration ----------------------------
  agent.config.cache.expiration: 30s

  kibana:
    enabled: true
    host: "https://elasticstack.phmic.com:5601"

    protocol: "https"
    username: "<elastic_user>"
    password: "<password>"

    ssl.enabled: true
    ssl.verification_mode: certificate
    ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
    ssl.certificate_authorities: 'D:\Elasticsearch\config\certauth-root.cer'
    ssl.certificate: 'D:\Elasticsearch\config\kibana-server.cer'
    ssl.key: 'D:\Elasticsearch\config\kibana-server.key'
  #---------------------------- APM Server - ILM Index Lifecycle Management ----------------------------
  ilm:
    enabled: "auto"

    setup:
      enabled: true
      overwrite: true
#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  hosts: ["https://elasticstack.mydomain.com:9200"]
  enabled: true
  compression_level: 0

  protocol: "https"
  username: "<elastic_user>"
  password: "<password>"
  worker: 1
  pipeline: "apm"
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
  ssl.certificate_authorities: 'D:\Elasticsearch\config\certauth-root.cer'
  ssl.certificate: 'D:\Elasticsearch\config\elastic-server.cer'
  ssl.key: 'D:\Elasticsearch\config\elastic-server.key'

[Trinity_Rolling]

-Delastic.apm.verify_server_cert=false

This setting did the trick! What does that do for our traffic between the APM Agent and server? Is this now treated and unencrypted traffic?

[riferrei (Ricardo Ferreira)]

This property essentially disables the APM server authentication, which includes SSL certificate validation on the client.

[stephenb (Stephen Brown)]

@Trinity_Rolling

This setting did the trick! What does that do for our traffic between the APM Agent and server? Is this now treated and unencrypted traffic?

The traffic is still encrypted just the server cert is just not validated on the client side.

[Trinity_Rolling]

I'll be honest. I've been struggling with this for a couple days now and this was my Hail Mary before giving up on Elastic. Thank you for your guidance. I still want a server side validation between the client and server, but I will need to involve our code team in order to touch the keystore of this application. Very early stages of a secured on-prem setup. Thank you again!

[riferrei (Ricardo Ferreira)]

We're happy to help you

[stephenb (Stephen Brown)]

The other way is just to make sure you have an official CA certificate then it should be fine I believe.

The other way and I'm not an expert at this is right it's get the CA from your internal certificate get it into the Java SSL key store and then it would probably work as well

[Trinity_Rolling]

I was thinking about this last evening and I think using the central CA should resolve this issue. Our development environment used a different CA for some reason or another. That flag is magic though.