How to define routing allocation of apm server index template

[Kay_Khan]

We would like to define the routing allocation of all index templates created by the apm server so that they are stored on our dedicated nodeset group "monitoring"

This is what we have attempted but unfortunately we find that the data gets stored on nodes other than the ones where the attri is "monitoring"

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server-prod
  namespace: elastic-system
spec:
  version: 8.7.1
  count: 1
  elasticsearchRef:
    name: "elasticsearch-prod"
  kibanaRef:
    name: "kibana-prod"
  http:
    service:
      spec:
        type: NodePort
  podTemplate:
    spec:
      containers:
      - name: apm-server
  config:
    http.enabled: true
    http.port: 5068
    monitoring.enabled: false
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
        anonymous:
          rate_limit:
            event_limit:  300
            ip_limit:  1000
          allow_service: ["admin-dashboard", "frontend"]
    apm-server.rum.enabled: true
    apm-server.rum.allow_origins: ["*"]
    apm-server:
      template.settings:
        index:
          routing.allocation.require.type: "monitoring"

[simitt (Silvia Mitter)]

The apm-server.template.* settings are not available anymore in 8.x, as the asset setup (index templates, component templates, ILM policies, etc.) moved to Kibana. Having multiple apm-servers with conflicting index setup configurations has caused problems in the past, therefore the setup is now centralized.
The templates are set up when installing the APM Integration. Once installed, @custom component templates available, where you can customize your settings.

[Kay_Khan]

I'm sorry i dont quite understand, Am i right in thinking this needs to be done via Kibana config?

This is what my kibana config looks like

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-prod
  namespace: elastic-system
spec:
  version: 8.7.1
  count: 1
  elasticsearchRef:
    name: elasticsearch-prod
  http:
    service:
      spec:
        type: NodePort
  podTemplate:
    metadata:
      labels:
        scrape: kb
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 4Gi
            cpu: 2
      nodeSelector:
        geeiq/node-type: ops
  config:
    monitoring.kibana.collection.enabled: false
    xpack.fleet.packages:
      - name: apm
        version: latest

Where should i be defining the

 apm-server:
      template.settings:
        index:
          routing.allocation.require.type: "monitoring"

Or where can i define the routing for these @custom templates in my yaml?

Also note the link you provided leads to an integration method for apm which i dont believe i'm using. Is this a problem, i'm quite confused at this point. As far as i understand my integration method is by defining. Taken from cloud-on-k8s/apm_es_kibana.yaml at main · elastic/cloud-on-k8s · GitHub

    xpack.fleet.packages:
      - name: apm
        version: latest

[simitt (Silvia Mitter)]

I'm sorry i dont quite understand, Am i right in thinking this needs to be done via Kibana config?

No unfortunately this can't be achieved via configuration. You can configure the custom component template via ES API calls.

Also note the link you provided leads to an integration method for apm which i dont believe i'm using. Is this a problem, i'm quite confused at this point.

The way you install the apm integration via kibana config is totally fine.

[Kay_Khan]

Okay thanks for confirming its no longer possible to codify these changes in our apm-server.yaml.

We have been following a standard when installing any elastic component and its a shame that we can't do this when it comes to the apm-server.

e.g filebeat and metricbeat look similar.

...
  config:
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"

---

We understand although we hope that you would take into consideration of allowing users to define the routing allocation via some .yaml configuration, so that we can codify the changes.

For now we have documented these api calls (which we will have to replicate for every apm component). Is there a section in the github that you can point us to do that we can refer to to see all the latest components ?

PUT _ilm/policy/apm-custom
{
  "policy": {
    "phases": {
      "hot": {
        "actions": { 
          "rollover": {
            "max_age": "30d",
            "max_size": "50GB",
            "max_primary_shard_size": "50gb"
            
          }
        }
      }
    }
  }
}

PUT _component_template/traces-apm.rum@custom
{
  "template": {
    "settings": {
      "routing": {
        "allocation": {
          "require": {
            "type": "monitoring"
          }
        }
      }, 
      "lifecycle": {
        "name": "apm-custom"
      }
    }
  }
}

GET _component_template/traces-apm.rum@custom

Although untested as we have seen that our datastream uses the *default_policy as opposed to the custom one. Do you know why that is?

[simitt (Silvia Mitter)]

Thanks for the feedback. We are having some internal discussions how we can improve the setup.

As for your concrete problem - please try a rollover on the datastream after changing the @custom component POST traces-apm.rum-default/_rollover.