APM agent deployment for rabbitmq

Elastic Observability APM
1

Hi Team,

We are trying to install apm agent for rabbitmq in kubernetes cluster to find the transactions. I have seen in the release notes that apm-agent-java versions greater than 1.20.0 supports APM for rabbitmq with java agent.

Can some one please share the statefulset deployment file for rabbitmq and apm-agent-java deployment.

Elasticsearch version : 7.12.0
Kibana version : 7.12.0
apm-agent-java : 7.12.0
apm-server : 7.12.0

Help is really appreciated. Thanks

2

Just install the latest Java agent (currently 1.23.0, very soon 1.24.0). The agent version doesn't need to match ELK or APM Server version. See docs.
If you are using it through Spring AMQP, you should use agent version 1.22.0.
As for RabbitMQ version, it needs to be 3.x - 5.x.
I hope this helps, not sure whether this is what you are looking for.

3

Thanks Eyal. Sorry for the delayed response.

We are using rabbit mq through Spring AMQP and I changes the agent version to 1.22.0 as suggested and rabbitmq version is 3.8.9.

Below is my yaml file. Correct me if am doing something wrong.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq-persistent
  namespace: env4
spec:
  serviceName: rabbitmq-persistent
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq-persistent
  template:
    metadata:
      labels:
        app: rabbitmq-persistent
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      ###################### Shared volume and init container ##########################
      volumes: 
      - name: elastic-apm-agent 
        emptyDir: {} 
      initContainers: 
      - name: elastic-java-agent 
        image: docker.elastic.co/observability/apm-agent-java:1.22.0 
        volumeMounts: 
        - mountPath: /elastic/apm/agent 
          name: elastic-apm-agent 
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
##################################################################################
      terminationGracePeriodSeconds: 10
      containers:
      - name: rabbitmq-persistent
        image: rabbitmq:3.8.9
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - >
                if [ -z "$(grep rabbitmq-persistent /etc/resolv.conf)" ]; then
                  sed "s/^search \([^ ]\+\)/search rabbitmq-persistent.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
                  cat /etc/resolv.conf.new > /etc/resolv.conf;
                  rm /etc/resolv.conf.new;

                  if [ -f "/etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf" ]; then
                    sed -i "s~management_agent.disable_metrics_collector = true~management_agent.disable_metrics_collector = false~g" /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf;
                    echo "cluster_partition_handling = pause_minority" >> /etc/rabbitmq/rabbitmq.conf;
                  fi;

                  rabbitmq-plugins --offline enable rabbitmq_peer_discovery_k8s
                  rabbitmq-plugins --offline enable rabbitmq_management;
                  rabbitmq-plugins --offline enable rabbitmq_tracing;

                  until rabbitmqctl node_health_check; do sleep 1; done;
                  if [ "$HOSTNAME" != "rabbitmq-persistent-0" ]; then
                    rabbitmqctl stop_app;
                    rabbitmqctl join_cluster rabbit@rabbitmq-persistent-0;
                    rabbitmqctl start_app;
                  fi;

                fi;

                rabbitmqctl add_user guest guest;

                rabbitmqctl set_policy ha-all "." '{"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic","ha-promote-on-shutdown":"when-synced","ha-promote-on-failure":"when-synced"}'
        env:
        - name: "JAVA_OPTS"
          value: "-Dspring.profiles.active=env"
        - name: ELASTIC_APM_SERVER_URLS
          value: "http://APM_server_ip:8200"
        - name: ELASTIC_APM_SERVICE_NAME
          value: rmq-apm
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: org.springframework.samples
        - name: ELASTIC_APM_ENVIRONMENT
          value: test
        - name: ELASTIC_APM_LOG_LEVEL
          value: DEBUG
        - name: JAVA_TOOL_OPTIONS
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
        ports:
        - containerPort: 5672
          name: amqp
        - containerPort: 25672
          name: rabbitmq-dist
        volumeMounts:
        - name: rabbitmq123
          mountPath: /var/lib/rabbitmq
        - name: elastic-apm-agent
          mountPath: /elastic/apm/agent
      volumes:
        - name: elastic-apm-agent
          emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq123
      annotations:
        volume.alpha.kubernetes.io/storage-class: default
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi # make this bigger in production

The deployment is successful and the pods are running and elastic-apm-agent.jar is copied to /elastic/apm/agent inside pod.

But am not receiving any data related to Rabbitmq.

Do I need to install java?
Please correct me if am doing some thing wrong.

4

Ohh, sorry about the confusion.
You cannot instrument the RabbitMQ server itself, it is not even written in Java AFAIK. I assumed you are looking for visibility into traces that write and read from RabbitMQ.

If your services are written in Java, you should apply the same configuration you did here onto your Java services, to get method level visibility of full traces that read-to/write-from RMQ. If your services are not Java, follow the proper installation instructions for the APM agent of your choice.

To enrich your tracing data with specific RMQ metrics, you can then use Metricbeat with its RabbitMQ module. Take a look at the docs explaining how to install on k8s, and about Autodiscover.

5

Hi Eyal,

Yes my services are written in java. I have added the apm-agent-java to the application and were able to see traces but that is not what am expecting.

Expected : To get the throughput and latency of each message that is processed by rabbitmq and all the messages information should be shown with service name rabbitmq.

Each message should be considered as transaction and latency should be calculated.

6

Sorry this is not what you expected.
Data about the RabbitMQ broker throughput can be monitored with other tools, including Metricbeat.

This is actually what the agent does on the receiver side. If you get distributed tracing working for you (meaning - the traces include both sending and receiving sides), then this already happens.

7

It requires installing java agent on all the java micro services to get the latency. I have 50 + applications which consumes messages from rabbitmq. How can i get all the latency in one place.

8

It sounds like you are using APM to get something that does not require APM. If you want to get deep visibility into your services/apps - APM is the tool for you and you can install an agent on any subset of your microservices.

If you want metrics about how your RabbitMQ is doing, you should use a proper tool for that (you know what I recommend already :slight_smile: )

9

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.