Is it possible to use the metadata processors for an apm server same as in the filebeat config

Elastic Observability APM
1

Is it possible to create indices for my apm agent like this using the processors add metadata option? And if its possible where do I adjust the settings? Have not found any example of a kubernetes yaml in the apm github repository.
Thanks a lot!

filebeatConfig:
      filebeat.yml: |
        processors:
        - add_cloud_metadata:
        - add_kubernetes_metadata:
            in_cluster: true
            matchers:
              - logs_path:
                  logs_path: "/var/lib/docker/containers/"

        output.elasticsearch:
          hosts: 'https://elasticsearch-master:9200'
          username: '${ELASTICSEARCH_USERNAME}'
          password: '${ELASTICSEARCH_PASSWORD}'
          index: "default-logs-%{[agent.version]}-rollover"
          indices:
          - index: "%{[kubernetes.namespace]:-}-k8s-logs-%{[agent.version]}-rollover"
            when.has_fields: ['kubernetes.namespace']
2

Yes, those processors are supported in apm-server too. You configure them just the same as in filebeat, only you put them in apm-server.yml instead of filebeat.yml.

Have not found any example of a kubernetes yaml in the apm github repository.

No, there isn't one in the repo, and I'm not sure where you can find one. Perhaps someone else can chime in. In case they're of use to you, there is a Helm chart, and Elastic Cloud on Kubernetes (ECK) supports APM Server orchestration.

1 Like
3

Hi @elk51211 Can you put the yaml of the apm-server? I can't make it work.

Thanks

4

Sure! That worked for me.

  apm-server.yml: |
    apm-server:
      host: "0.0.0.0:8200"
    setup:
      template:
        name: "apm-%{[observer.version]}"
        pattern: "*-apm-%{[observer.version]}-*"
    
    # add processors to get metadata on kubernetes namespaces
    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          in_cluster: true
          matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/"
    queue: {}
    output.elasticsearch:
      username: '${ELASTICSEARCH_USERNAME}'
      password: '${ELASTICSEARCH_PASSWORD}'
      index: "default-apm-%{[observer.version]}"
      indices:
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-sourcemap"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "sourcemap"
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "error"
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "transaction"
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "span"
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "metric"
        - index: "%{[kubernetes.namespace]:-}-apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
          when.has_fields: ['kubernetes.namespace']
          when.contains:
            processor.event: "onboarding"
      protocol: https
      hosts: ["elasticsearch-master:9200"]
      ssl.certificate_authorities:
        - /usr/share/apm-server/config/certs/elastic-certificate.pem
secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/apm-server/config/certs

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
5

Thanks @elk51211.

In my case is not working because I have Dynamic index setting to false.

Official doc say that dynamic setting is defined as true but i do not defined it in any place. Anyone know how to set it in the yaml config?

Thanks ^^

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.