# Split traces to separate indices based on service name

**URL:** <https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942>\
**Category:** APM\
**Tags:** docker, nodejs\
**Created:** [January 31, 2025, 1:26pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942 "2025-01-31T13:26:11Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ali\_Osman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_osman/32/141071_2.png) [@Ali\_Osman](https://discuss.elastic.co/u/Ali_Osman)\
**Post date:** [January 31, 2025, 1:26pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/1 "2025-01-31T13:26:11Z")

</div>

**Kibana version** : 8.17.1

**Elasticsearch version** : 8.17.1

**APM Server version** : 8.17.1

**APM Agent language and version** : NodeJS - 4.11.0

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Docker

**Fresh install or upgraded from other version?**  
Fresh

Essentially what I want to do is, I have a couple of node servers called service 1 & 2 I have the agents set up in them as so

```auto
const apm = require('elastic-apm-node').start({
    serviceName: 'node-test [1,2]',
    secretToken: 'supersecrettoken',
    serverUrl: 'http://localhost:8200',
    captureBody: 'all',
});

```

And they log to a hidden index called `.ds-traces-apm-default` but I got like 60k documents. When I filtered with the service name `node-test` I got 88 document.

My question is can I split the requests being sent to the server to a separate index that I can filter on and make dashboards on and set each service in a different index ?

This is the docker compose I use

````auto
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  fleetserverdata:
    driver: local

networks:
  default:
    name: elastic
    external: false

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    # " - name: es02\n"\
    # " dns:\n"\
    # " - es02\n"\
    # " - localhost\n"\
    # " ip:\n"\
    # " - 127.0.0.1\n"\
    # " - name: es03\n"\
    # " dns:\n"\
    # " - es03\n"\
    # " - localhost\n"\
    # " ip:\n"\
    # " - 127.0.0.1\n"\
    command: >
      bash -c '
        if [x${ELASTIC_PASSWORD} == x]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [x${KIBANA_PASSWORD} == x]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [! -f config/certs/ca.zip]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [! -f config/certs/certs.zip]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          " - name: es01\n"\
          " dns:\n"\
          " - es01\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: kibana\n"\
          " dns:\n"\
          " - kibana\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: fleet-server\n"\
          " dns:\n"\
          " - fleet-server\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[-f config/certs/es01/es01.crt]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      # - cluster.initial_master_nodes=es01,es02,es03
      # - discovery.seed_hosts=es02,es03
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=config/certs/kibana/kibana.crt
      - SERVER_SSL_KEY=config/certs/kibana/kibana.key
      - SERVER_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -I -s --cacert config/certs/ca/ca.crt https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  fleet-server:
    depends_on:
      kibana:
        condition: service_healthy
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/certs
      - fleetserverdata:/usr/share/elastic-agent
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    ports:
      - ${FLEET_PORT}:8220
      - ${APMSERVER_PORT}:8200
    user: root
    environment:
      - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - FLEET_CA=/certs/ca/ca.crt
      - FLEET_ENROLL=1
      - FLEET_INSECURE=true
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
      - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
      - FLEET_SERVER_INSECURE_HTTP=true
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - FLEET_URL=https://fleet-server:8220
      - KIBANA_FLEET_CA=/certs/ca/ca.crt
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_USERNAME=elastic
      - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_HOST=https://kibana:5601```

````

Thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 31, 2025, 2:43pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/2 "2025-01-31T14:43:14Z")

</div>

Hi @Ali_Osman, Welcome to the community. Nice Compose!

> [@Ali\_Osman](#):
>
> My question is can I split the requests being sent to the server to a separate index that I can filter on and make dashboards on and set each service in a different index ?

This is not a really a good idea and is considered an anti-pattern. I would recommend against it. If you do this, you will end up with many small data streams / indices, which can have an adverse effects.

I also recommend against this if you are new... Elastic Provides solid defaults / architecture OOTB I would run it for a while before you try to make changes like this.

You can easily build separate dashboards by simply filtering on `service.name` or you can build a single dashboard and use the drop down filter control to just select the service you are looking for.

I have users with 100s of services traces in the single data stream... works just fine.

And of course there is the built in APM UI which does this for you.

Perhaps start with that before try changes that is not that easy and can have negative effects....

---

<div class="post-metadata">

**Author:** ![Ali\_Osman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_osman/32/141071_2.png) [@Ali\_Osman](https://discuss.elastic.co/u/Ali_Osman)\
**Post date:** [January 31, 2025, 11:39pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/3 "2025-01-31T23:39:30Z")

</div>

Okay, Thank you for the quick response

I have one more question.

the index itself becomes flooded with documents from kibana because it becomes an apm agent from the get go. Is there a way to keep kibana out of the index or stop it from logging to the index so I only have the node services in the index ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 1, 2025, 1:10am UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/4 "2025-02-01T01:10:02Z")

</div>

> [@Ali\_Osman](#):
>
> the index itself becomes flooded with documents from kibana because it becomes an apm agent from the get go.

Ok, let's back up a bit...

Are you sending Kibana APM traces to the same Cluster that you are doing APM / traces for your Business Services? (which is kinda ok I guess but a bit unusual, typically all monitoring of Elastic Components go to another Monitoring Cluster....

What do you mean

> "kibana because it becomes an apm agent from the get go."

Why are you enabling Traces on Kibana in the first place?

Are you telling me you are doing this by adding this to your kibana.yml

Per [this article](https://www.elastic.co/blog/apm-kibana-elasticsearch-alerting-insights)

```auto
elastic.apm.active: true
elastic.apm.serverUrl: url of the APM server
elastic.apm.secretToken: Token that the APM server exposes

```

I am pretty confused, I must be missing something ...

---

<div class="post-metadata">

**Author:** ![Ali\_Osman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_osman/32/141071_2.png) [@Ali\_Osman](https://discuss.elastic.co/u/Ali_Osman)\
**Post date:** [February 1, 2025, 12:53pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/5 "2025-02-01T12:53:18Z")

</div>

You're correct I overlooked it.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 1, 2025, 3:14pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/6 "2025-02-01T15:14:01Z")

</div>

Hi @Ali_Osman

What exactly did you overlook?  
Perhaps you can provide a little bit of detail so if someone else runs into the same problem they can understand what you did to fix  
Thank you!

---

<div class="post-metadata">

**Author:** ![Ali\_Osman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_osman/32/141071_2.png) [@Ali\_Osman](https://discuss.elastic.co/u/Ali_Osman)\
**Post date:** [February 1, 2025, 4:26pm UTC](https://discuss.elastic.co/t/split-traces-to-separate-indices-based-on-service-name/373942/7 "2025-02-01T16:26:44Z")

</div>

I did overlook that I had these values set

```auto
elastic.apm.active: true
elastic.apm.serverUrl: url of the APM server
elastic.apm.secretToken: Token that the APM server exposes

```

In the kibana yml file