Unable to start APM Java Agent in WAS K8s pods

Elastic Observability APM
1

Kibana version: 8.5.2

Elasticsearch version: 8.5.2

APM Server version: 7.13.0

APM Agent language and version: apm-agent-java:1.12.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
https://www.elastic.co/es/blog/using-elastic-apm-java-agent-on-kubernetes-k8s
" ### Install without modifying the application image" Method

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

No, as far as I know

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I've just applied a K8s manifesto to get a microservice app that run 3 java pods in WebSphere Application Servers. Allthough the volume with the agent is mounted I cant see in the pod's logs any agent running.
Is WAS compatible with this agent?
Is there any particular method to install it in WAS server

Thank you in advance and best regards.

Carlos T

2

Hi and thanks for asking.

How come you are using 1.12.0? It is extremely old, best if you try the latest version.

We do not officially support WAS, but this is mostly because we do not regularly test with it. We do test regularly with WebSphere Liberty though and we know there are plenty of users that use it on WAS with success, so it should work as far as I know.

Mounting that agent jar on the pod is not enough, a -javaagent instruction needs to be added somehow to the command line. We have a blog post that explains how this can be achieved. Even if it is a bit outdated, it still provide useful info.

What this blog post ends up proposing is relying on the JAVA_TOOL_OPTIONS environment variable to append -javaagent to the command line. We now have a new k8s APM attacher, still in technical preview, that does exactly that automatically by relying on the k8s mutating webhook technology. Once you realize how to use it, it would be by far the

Please go through all this material and hopefully you will figure this out.

3

Thank you very much Eyal.

I tried to apply this method to install the Java agent but I can only download the docker image and I dont know hot to modify it as I dont have the initial .yaml file and I lack the knowledge and time to investigate this as I've been requested to attend to different duties.

Anyway. I take good note of your advice and I will follow it I get the chance.

Thanks a lot for you time and help.

Carlos T.

4

You don't need to know anything about the agent image. If you followed the blog post I referred to above, you should end up without any change in the app or agent images, only a pod configuration within the manifest that looks something like that:

apiVersion: ... 
kind: .... 
metadata: 
  name: .... 
  namespace: ... 
spec: 
  replicas: .. 
  ...
      ############# Shared volume and init container ##################
      volumes: 
      - name: elastic-apm-agent 
        emptyDir: {} 
      initContainers: 
      - name: elastic-java-agent 
        image: docker.elastic.co/observability/apm-agent-java:latest
        volumeMounts: 
        - mountPath: /elastic/apm/agent 
          name: elastic-apm-agent 
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      ####################################################################################      
      containers: 
      - name: ... 
        image: ....
      ################### Volume path and agent config ################
        volumeMounts: 
        - mountPath: /elastic/apm/agent 
          name: elastic-apm-agent 
        env: 
        - name: ELASTIC_APM_SERVER_URL 
          value: "http://<my-apm-server-url>" 
        - name: ELASTIC_APM_SERVICE_NAME 
          value: "myapp" 
        - name: ELASTIC_APM_APPLICATION_PACKAGES 
          value: "...." 
        - name: ELASTIC_APM_ENVIRONMENT 
          value: ... 
        - name: ELASTIC_APM_SECRET_TOKEN 
          valueFrom: 
            secretKeyRef: 
              name: apm-server-apm-token 
              key: secret-token 
        - name: JAVA_TOOL_OPTIONS 
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
      #############################################################

In order to monitor services running on k8s pods you would need to either change their manifest, or to change the app images themselves and you definitely need to put at least some time in it :slightly_smiling_face:

Good luck

5

Thank you very much Eyal.

I've tried so. I've worked with a simple Java hello world pod app.

But Im trying this one:

https://github.com/digitalocean/kubernetes-sample-apps/tree/master/bookinfo-example

It creates a polyglot app with different pods runnings. Three of them named 'reviews-v1', 'reviews-v2', 'reviews-v3' that run Java apps in WAS.

Im trying so far, to get the APM agent running in the 'reviews-v1' pod.

The service originally looked like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.4
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}

And I have modified it to look like this:

apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.4
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        - name: elastic-apm-agent
          mountPath: /elastic/apm/agent
        securityContext:
          runAsUser: 1000
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
      - name: elastic-apm-agent
        emptyDir: {}
      initContainers:
      - name: elastic-java-agent
        image: docker.elastic.co/observability/apm-agent-java:1.12.0
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      ######################### Volume path and agent config ########################### AQUI COMIENZA SEGUNDO AGREGADO
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        env:
        - name: ELASTIC_APM_SERVER_URL
          value: "http://apm-server.eck.svc.cluster.local:8200"
        - name: ELASTIC_APM_SERVICE_NAME
          value: "bookinfo"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "bookinforeview2"
        - name: ELASTIC_APM_ENVIRONMENT
          value: test
        - name: ELASTIC_APM_LOG_LEVEL
          value: DEBUG
        - name: ELASTIC_APM_SECRET_TOKEN
          value: gincol
        - name: JAVA_TOOL_OPTIONS
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar

The yaml is much bigger but I want to focus in this service.

I know that the env values work as I've set the clinic vet app example running it in a different namespace by using this settings.

But once the app is deployed if I review the logs of the pod:

kubectl logs reviews-v1-7d4f5d97b5-br6bk -n bookinfo
Defaulted container "reviews" out of: reviews, elastic-java-agent (init)

Launching defaultServer (WebSphere Application Server 20.0.0.6/wlp-1.0.41.cl200620200528-0414) on IBM J9 VM, version 8.0.6.16 - pxa6480sr6fp16-20200902_01(SR6 FP16) (en_US)
[AUDIT ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: Software License
[AUDIT ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT ] CWWKT0016I: Web application available (default_host): http://reviews-v1-7d4f5d97b5-br6bk:9080/
[AUDIT ] CWWKZ0001I: Application reviews-app started in 0.275 seconds.
[AUDIT ] CWWKF0012I: The server installed the following features: [jaxrs-2.0, jaxrsClient-2.0, json-1.0, jsonp-1.0, servlet-3.1].
[AUDIT ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 4.024 seconds.

I know that volume with the agent is mounted, as I ssh the pod to check it
And this is where I'm stuck.

So if someone please knows hot to deal with the 'Defaulted container "reviews" out of: reviews, elastic-java-agent (init' error, that would make me happy.

Anyway, thank you very much just for trying. I appreciate it.

Regards.

Carlos T.

6

Try this instead:

apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
    spec:
      serviceAccountName: bookinfo-reviews
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
      - name: elastic-apm-agent
        emptyDir: {}
      initContainers:
      - name: elastic-java-agent
        image: docker.elastic.co/observability/apm-agent-java:latest
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      containers:
      - name: reviews
        image: docker.io/istio/examples-bookinfo-reviews-v1:1.16.4
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        - name: ELASTIC_APM_SERVER_URL
          value: "http://apm-server.eck.svc.cluster.local:8200"
        - name: ELASTIC_APM_SERVICE_NAME
          value: "bookinfo"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "bookinforeview2"
        - name: ELASTIC_APM_ENVIRONMENT
          value: test
        - name: ELASTIC_APM_LOG_LEVEL
          value: DEBUG
        - name: ELASTIC_APM_SECRET_TOKEN
          value: gincol
        - name: JAVA_TOOL_OPTIONS
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
        - name: elastic-apm-agent
          mountPath: /elastic/apm/agent
        securityContext:
          runAsUser: 1000

The order is probably not important, I only reordered it so that it is more readable. The important change I did is setting these env variables to the actual application container, rather than the init container. The init container does nothing, it only contains the agent jar and then the cp command copies it into the shared volume elastic-apm-agent, so that it is visible to the app container. Then, the JAVA_TOOL_OPTIONS tells the JVM in the app container to append the -javaagent setting to the app Java command. This and all other env variables are important for the traced container, not the init container.

I did not try this manifest, so sorry if something in it is messed up. If so, just apply the required change - move the env variables to the actual container.

I hope this helps.

7

Yes Sir!!

Indeed it worked as a charm :).

Thank you very much, not just for the time to make readible, for the fix, and mostly for the knowledge and explanation that allowed me to understand what wasn't working well.

This app uses the Java pods calle review1, review2 and review3

So far I only installed the APM in the review1.

If I want to make it in the other two I have a doubt about how to classify it in a logical way

Should all them be

For the pod review1

- name: ELASTIC_APM_SERVICE_NAME
          value: "bookinfo"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "bookinforeview1"

For the pod review2

- name: ELASTIC_APM_SERVICE_NAME
          value: "bookinfo"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "bookinforeview2"

For the pod review3

- name: ELASTIC_APM_SERVICE_NAME
          value: "bookinfo"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "bookinforeview3"

They all belong to the same app.

Thank you very much again Eyal. You make an other person happy today.

1 Like
8

Thanks a lot for the feedback! It made be happy as well :blush:

I highly encourage you to go thoroughly over our documentation, especially for these types of questions, it will allow you the get the most out of our offering.

service_name is the major high-level aggregation for APM traces. If your pods are running difference services, then each should get its own service_name.
If each of these pod runs the exact same service logic and you use multiple instances in order to get horizontal scalability, then all should have the same service name but each would have a different service_node_name. You don't have to manually set it though, only if you want it to get a specific name. If you don't set it, the service node name would be the container ID.
As for application_packages- this is a completely irrelevant configuration. It should be set with the root package name of your application classes.

I hope this helps. Good luck!

9

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.