Problems with ssl apm java agent to elastic cloud

irivas95 · January 25, 2023, 10:29am

HI,
I am trying to deploy an application with an APM java agent in GKE that sends data to an APM integration in elastic cloud.
I am using this Dockerfile

# vi Dockerfile
FROM adoptopenjdk:11-jre-hotspot
RUN export
RUN apt-get -qq update \
   && apt-get install --no-install-recommends -y -qq curl \
   && rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true
WORKDIR /app
COPY . ./
ENV JAVA_OPTS="-Xmx512m"
LABEL \
      org.label-schema.schema-version="1.0" \
      org.label-schema.vendor="Elastic" \
      org.label-schema.name="opbeans-java" \
      org.label-schema.version="1.18.1" \
      org.label-schema.url="https://hub.docker.com/r/opbeans/opbeans-java" \
      org.label-schema.vcs-url="https://github.com/elastic/opbeans-java" \
      org.label-schema.license="MIT"

CMD java $JAVA_OPTS -javaagent:/app/elastic-apm-agent-1.30.0.jar -Delastic.apm.service_name=opbeans-java-k8s -Delastic.apm.secret_token=APM_SECRET_TOKEN -Delastic.apm.server_url=https://apm_url:443 -Delastic.apm.environment=production -Delastic.apm.application_packages=co.elastic.apm.opbeans  -Dspring.profiles.active=${OPBEANS_JAVA_PROFILE:-} -Dserver.port=${OPBEANS_SERVER_PORT:-} -Dserver.address=${OPBEANS_SERVER_ADDRESS:-0.0.0.0} -Dspring.datasource.url=${DATABASE_URL:-} -Dspring.datasource.driverClassName=${DATABASE_DRIVER:-} -Dspring.jpa.database=${DATABASE_DIALECT:-} -jar /app/opbeans-0.0.1-SNAPSHOT.jar

and the following deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opbeans-java
  labels:
     app: openbeans-java
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opbeans-java
  template:
    metadata:
      labels:
        app: opbeans-java
    spec:
      containers:
      - name: opbeans-java
        # Replace $LOCATION with your Artifact Registry location (e.g., us-west1).
        # Replace $GCLOUD_PROJECT with your project ID.
        image: openbeans/opbeans-java-cloud:latest
        # This app listens on port 8080 for web traffic by default.
        ports:
        - containerPort: 8080
        env:
          - name: ELASTIC_APM_ENABLE_LOG_CORRELATION
            value: "true"
          - name: ELASTIC_APM_VERIFY_SERVER_CERT
            value: "false"

it seems that the agent connects to my elastic cloud at some point, as I manage to receive traces, but i get these errors

2023-01-25 10:13:08,230 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
11:13:08.233
2023-01-25 10:13:08,233 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified
11:13:08.237
2023-01-25 10:13:08,233 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null
11:13:08.238
2023-01-25 10:13:08,238 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at https://apm_url:443/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.
11:13:08.239
2023-01-25 10:13:08,239 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Default cipher suites: [TLS_AES_128_GCM_SHA256....]

does anyone know why these errors may be appearing?

Thanks in advance

Jonas_Kunz · January 25, 2023, 11:09am

Hi @irivas95 ,

The ELASTIC_APM_VERIFY_SERVER_CERT=false configuration should not be required and not be used when connecting to an elastic cloud APM server, as those should have proper TLS certificates out of the box.

Your docker image contains the following connection configuration in the CMD part:

-Delastic.apm.secret_token=APM_SECRET_TOKEN -Delastic.apm.server_url=https://apm_url:443

I'm not sure whether you redacted those for posting here or whether you forget to configure these values correctly. Please check that you actually use the values presented to you by the APM integration in elastic cloud (the redacted options in my screenshot):

irivas95 · January 25, 2023, 11:17am

Hi @Jonas_Kunz,
Thanks for your reply.
Sorry for not commenting it explicitly, the values that you comment if they are configured with the data that appear in the APM integration, I have published them this way for security reasons.
As for the deployment, I also tested it without the option ELASTIC_APM_VERIFY_SERVER_CERT=false, but I get the same errors.

Jonas_Kunz · January 25, 2023, 11:24am

That's strange. Could you provide us the full agent debug logs, this should hopefully give us a bit more information on what might be wrong.
You can use GitHub gists for uploading the log.

irivas95 · January 25, 2023, 12:21pm

with debug I only receive this error in addition to those mentioned above

java.net.UnknownHostException: apm.us-central1.gcp.cloud.es.io at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?] 
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?] at java.net.Socket.connect(Unknown Source) ~[?:?] at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:?] 
at co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket.connect(TLSFallbackSSLSocket.java:219) ~[elastic-apm-agent-1.30.0.jar:?] at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:?]
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?] at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?] at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source) ~[?:?] 
at sun.net.www.protocol.https.HttpsClient.New(Unknown Source) ~[?:?] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source) ~[?:?] 
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:?] 
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) ~[?:?] at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source) ~[?:?] 
at co.elastic.apm.agent.report.AbstractIntakeApiHandler.startRequest(AbstractIntakeApiHandler.java:113) ~[elastic-apm-agent-1.30.0.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:170) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleIntakeEvent(IntakeV2ReportingEventHandler.java:117) [elastic-apm-agent-1.30.0.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.dispatchEvent(IntakeV2ReportingEventHandler.java:103) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:70) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:36) [elastic-apm-agent-1.30.0.jar:?] 
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.30.0.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.30.0.jar:?] at java.lang.Thread.run(Unknown Source) [?:?]

Jonas_Kunz · January 25, 2023, 12:49pm

It looks like your APM server is not reachable from within your container (e.g. due to a firewall).
You can check this by SSHing into one of your containers in your GKE environment and executing a curl to the APM server url. You will need to check the networking settings of your environment and ensure that the APM server is accessible from there.

irivas95 · January 25, 2023, 3:22pm

I have opened a test firewall rule allowing outbound to any destination and still get the same errors and if I launch a curl to the url from the gke node where the pod is located I get no response.
If I launch the curl to the public IP of the APM integration (I get it with nslookup) I receive the following error:

{"ok":false,"message":"Unknown resource."}

Jonas_Kunz · January 26, 2023, 9:11am

Running curl against the APM-server url should give you a 200 response with no body.

E.g. if I run curl -i https://<redacted>.apm.europe-west3.gcp.cloud.es.io:443
I get the following result:

HTTP/2 200 
date: Thu, 26 Jan 2023 09:05:35 GMT
x-cloud-request-id: euac0uJuRdef9FHXNV8PAQ
x-content-type-options: nosniff
x-found-handling-cluster: <redacted>
x-found-handling-instance: instance-0000000007
content-length: 0

To exclude network configuration problems you must run this command from the very same environment where your Java application is running. E.g. extend your Dockerfile CMD to execute the curl-command before starting your App.

irivas95 · January 26, 2023, 1:12pm

thank you for your involvement Jonas.
I have performed the following tests:

test 1:

 kubectl exec -it opbeans-java -- curl -X POST https://xxxxxxxxxxx.apm.us-central1.gcp.cloud.es.io:443/ \
  -H "Authorization: Bearer XXXXXXXXX"

response:

{
  "build_date": "2023-01-03T05:32:18Z",
  "build_sha": "12fb981fea24575b96066ab699fd7ab7cea820c2",
  "publish_ready": true,
  "version": "8.6.0"
}

test 2:

kubectl exec -it opbeans-java -- curl -i -k -XPOST https://xxxxxxxxxxx.apm.us-central1.gcp.cloud.es.io:443 -H "Authorization: Bearer XXXXXXXX"

response:

HTTP/2 404
content-type: application/json; charset=UTF-8
x-cloud-request-id: bzY610daQFi0p0RMBFl6Rg
content-length: 43
date: Thu, 26 Jan 2023 12:54:46 GMT

test 3

kubectl exec -it opbeans-java -- curl -i -k -XPOST https://PUBLIC_IP:443 -H "Authorization: Bearer XXXXXXXX"

response:

HTTP/2 404
content-type: application/json; charset=UTF-8
x-cloud-request-id: bzY610daQFi0p0RMBFl6Rg
content-length: 43
date: Thu, 26 Jan 2023 12:54:46 GMT

test 4:

kubectl exec -it opbeans-java -- curl -i -k https://PUBLIC_IP:443 -H "Authorization: Bearer XXXXXXXX"

response:

HTTP/2 404
content-type: application/json; charset=UTF-8
x-cloud-request-id: RtydSn3WTjOsPxPwm8oz7g
content-length: 43
date: Thu, 26 Jan 2023 12:54:22 GMT

{"ok":false,"message":"Unknown resource."}

Jonas_Kunz · January 27, 2023, 2:06pm

Okay, your test1 shows that the apm-server is reachable from your container.
I think it makes sense that the IP-based tests fail because we (most likely) don't allocate a separate IP & proxy per cloud account.

I have a question regarding the debug logs you posted earlier:

java.net.UnknownHostException: apm.us-central1.gcp.cloud.es.io at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?] 
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?] at java.net.Socket.connect(Unknown Source) ~[?:?] at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:?]

In the logs it shows apm.us-central1.gcp.cloud.es.io, which is wrong. I think it should be xxx.apm.us-central1.gcp.cloud.es.io where xxx is the ID of your cloud deployment.

Did you just redact this and it actually prints the hostname including your cloud id? Otherwise I think the agent might be picking up your configuration wrongly.
To debug that, I would need the full debug logs, including the startup. Please just redact the secrets / your deployment id with xxxx like you did in your last post.

irivas95 · January 30, 2023, 10:24am

Hi @Jonas_Kunz ,
sorry for the delay.
these are the logs:

2023-01-30 10:13:56,678 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - APM Server identity could not be verified

2023-01-30 10:13:56,678 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket - Local certificates: null

2023-01-30 10:13:56,679 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at https://XXXXXXXXXXX.apm.us-central1.gcp.cloud.es.io:443/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.

2023-01-30 10:13:56,679 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Default cipher suites: [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2023-01-30 10:13:56,679 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Supported cipher suites: [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

2023-01-30 10:13:56,679 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: XXXXXXXXXXXX.apm.us-central1.gcp.cloud.es.io

2023-01-30 10:13:56,679 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Event handling failure
java.net.UnknownHostException: XXXXXXXXXXXXXX.apm.us-central1.gcp.cloud.es.io at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?] 
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?] at java.net.Socket.connect(Unknown Source) ~[?:?] at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:?] 
at co.elastic.apm.agent.report.ssl.TLSFallbackSSLSocket.connect(TLSFallbackSSLSocket.java:219) ~[elastic-apm-agent-1.30.0.jar:?] at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:?] 
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?] at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:?] at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source) ~[?:?] 
at sun.net.www.protocol.https.HttpsClient.New(Unknown Source) ~[?:?] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source) ~[?:?] 
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:?] 
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) ~[?:?] at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source) ~[?:?] 
at co.elastic.apm.agent.report.AbstractIntakeApiHandler.startRequest(AbstractIntakeApiHandler.java:113) ~[elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:170) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleIntakeEvent(IntakeV2ReportingEventHandler.java:117) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.dispatchEvent(IntakeV2ReportingEventHandler.java:103) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:70) [elastic-apm-agent-1.30.0.jar:?] 
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:36) [elastic-apm-agent-1.30.0.jar:?] 
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.30.0.jar:?] at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.30.0.jar:?] at java.lang.Thread.run(Unknown Source) [?:?]

creastysomp · January 30, 2023, 11:02am

I have opened a test firewall rule allowing outbound to any destination and still get the same errors and if I launch a curl to the url from the gke node where the pod is located I get no response.

Jonas_Kunz · January 30, 2023, 11:18am

Hi @irivas95 the URL in your provided log seems correct.
Nonetheless, we would need the full agent debug logs including the startup to further help you here, otherwise I'm just guessing.

irivas95 · January 30, 2023, 3:31pm

these are logs of the agent startup, I hope they can be of some help

[main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
[main] DEBUG co.elastic.apm.agent.metrics.builtin.CGroupMetrics - max cgroup memory read from /sys/fs/cgroup/memory/memory.limit_in_bytes is: 9223372036854771712
[main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific ThreadMXBean interface: com.sun.management.ThreadMXBean
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.35.0 as opbeans-java-k8s (0.0.1-SNAPSHOT) on Java 11.0.11 Runtime version: 11.0.11+9 VM version: 11.0.11+9 (AdoptOpenJDK) Linux 5.10.147+
[main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-Xmx512m, -javaagent:/app/elastic-apm-agent-1.35.0.jar, -Delastic.apm.service_name=opbeans-java-k8s, -Delastic.apm.secret_token=XXXXXXXXX, -Delastic.apm.server_url=https://XXXXXXXXX.apm.us-central1.gcp.cloud.es.io:443, -Delastic.apm.environment=production, -Delastic.apm.application_packages=co.elastic.apm.opbeans, -Dspring.profiles.active=, -Dserver.port=, -Dserver.address=0.0.0.0, -Dspring.datasource.url=, -Dspring.datasource.driverClassName=, -Dspring.jpa.database=]
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'opbeans-java-k8s' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - environment: 'production' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://XXXXXXXX.apm.us-central1.gcp.cloud.es.io:443' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'co.elastic.apm.opbeans' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - log_level: 'DEBUG' (source: Environment Variables)
[main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - No plugins dir
[elastic-apm-metadata-1] WARN  co.elastic.apm.agent.impl.metadata.CloudMetadataProvider - Unexpected error during automatic discovery process for cloud provider
java.util.concurrent.TimeoutException: null
        at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
        at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.tryAllCloudProviders(CloudMetadataProvider.java:190) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.fetchAndParseCloudProviderInfo(CloudMetadataProvider.java:129) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at co.elastic.apm.agent.impl.metadata.CloudMetadataProvider.getCloudInfoProvider(CloudMetadataProvider.java:81) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at co.elastic.apm.agent.impl.metadata.MetaData$2.call(MetaData.java:129) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at co.elastic.apm.agent.impl.metadata.MetaData$2.call(MetaData.java:125) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.35.0.jar:1.35.0]
        at java.lang.Thread.run(Unknown Source) [?:?]
[main] DEBUG co.elastic.apm.agent.util.ExecutorUtils - A new thread named `elastic-apm-type-cache-pool-cleaner` was created. The original context class loader of this thread (jdk.internal.loader.ClassLoaders$AppClassLoader@57db2b13) has been overridden

Jonas_Kunz · January 31, 2023, 9:34am

Sorry, we need the entire log file. Please follow this procedure and upload the entire log file, otherwise we can't assist you any further.

creastysomp · February 13, 2023, 10:44am

I have opened a test firewall rule allowing outbound to any destination and still get the same errors.

.

Jack_Shirazi · February 13, 2023, 4:24pm

Per what Jonas said, we'll need the entire log to get any further

system · March 6, 2023, 12:25pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.