# Create custom data streams per service **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [November 8, 2023, 1:52pm UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710 "2023-11-08T13:52:06Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Namita\_Jaokar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/namita_jaokar/32/104106_2.png) [@Namita\_Jaokar](https://discuss.elastic.co/u/Namita_Jaokar)\ **Post date:** [November 8, 2023, 1:52pm UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/1 "2023-11-08T13:52:06Z")
Hi, Is there any way I can create custom data stream per service in with elastic agent.? I came across few solutions in the discussions , But probably its not been supported in version 8.6.1 or 8.x above. Below is the solution that was being tried out: > [@Storing APM data in custom/specific indices](https://discuss.elastic.co/t/storing-apm-data-in-custom-specific-indices/246381): > > If you'd like to split your APM indices by the service, you can use the advice from [this topic](https://discuss.elastic.co/t/how-to-create-index-for-each-service/242821). You can change the index name by setting either [output.elasticsearch.index](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#index-option-es) or [output.elasticsearch.indices](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#indices-option-es). These configuration variables take an index name "format string", which can reference fields the events. For your use-case, you can include %{[service.name]} in the format string to create an index per service. e.g. output.elasticsearch: hosts: ["http://localhost:9200"] indices: - inde… But seems above solution is not working. Can someone please advice on this. Is it possible to create data streams for respective services I want to maintain the transaction traces separately for every elastic agent. Best Regards, Namita Jaokar ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [November 13, 2023, 6:27am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/2 "2023-11-13T06:27:51Z")
@Namita_Jaokar sorry for the delay in responding. You're right that this won't work in newer versions of APM Server, as we no longer support `output.elasticsearch.index` or `output.elasticsearch.indices`. These days, APM Server will always write data to specific data streams. If you wish to route data to more specific data streams, you can do this in Elasticsearch using an ingest pipeline. See [[OpenTelemetry] data\_stream.namepace and data\_stream.datastream aren't being respected · Issue #10191 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/10191#issuecomment-1707776700) ---
**Author:** ![Namita\_Jaokar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/namita_jaokar/32/104106_2.png) [@Namita\_Jaokar](https://discuss.elastic.co/u/Namita_Jaokar)\ **Post date:** [November 17, 2023, 10:10am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/3 "2023-11-17T10:10:27Z")
Hi @axw , Thank you for your response. I did check some links related to ingest pipeline as its a new concept for me to begin with. I went through below link to understand the ingest pipeline > **[Parse data using ingest pipelines | APM User Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/apm/guide/8.6/ingest-pipelines.html#custom-ingest-pipeline-create)** _Sending other APM data to alternate data streams, like traces (`traces-apm.*` ), logs (`logs-apm.*` ), and internal metrics (`metrics-apm.internal*` ) is not currently supported._ This was mentioned in the documentation of ingest pipeline. Does that mean we cannot push apm-traces to a another/custom datastream. Also, I tried to create a pipeline using GUI under stack management section. For the same, We need to create a processor. When I try to add processor , It gives me below options ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/7/47b47d794b3c33665a9e5a25e2b761068aee4768.png) With which of the above options do I need to ask elasticsearch to create a new data stream for each of the service.name that are running. Please correct me if my understanding is wrong. Thanks in Advance, Namita Jaokar ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [November 20, 2023, 1:56am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/4 "2023-11-20T01:56:52Z")
> [@Namita\_Jaokar](#): > > _Sending other APM data to alternate data streams, like traces (`traces-apm.*` ), logs (`logs-apm.*` ), and internal metrics (`metrics-apm.internal*` ) is not currently supported._ What this means is you can't for example send traces to a metrics data stream, or vice versa. This section needs a bit of tidying up, so I'll see if we can clarify this statement when we do. > With which of the above options do I need to ask elasticsearch to create a new data stream for each of the service.name that are running. Sorry, I missed earlier that you were using 8.6. You will need to use a more recent version of the stack (8.10.x) so you can use the "reroute" ingest processor. ---
**Author:** ![Namita\_Jaokar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/namita_jaokar/32/104106_2.png) [@Namita\_Jaokar](https://discuss.elastic.co/u/Namita_Jaokar)\ **Post date:** [November 20, 2023, 2:18pm UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/5 "2023-11-20T14:18:10Z")
Thanks @axw for your help. Will check if its within the scope of my requirement to upgrade to 8.10 or above. Just wanted to confirm one more thing, I should be able to implement ingest processor specifically of type "reroute" with open source or do I need to have enterprise level subscription for it. Best Regards, Namita Jaokar ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [November 21, 2023, 2:59am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/6 "2023-11-21T02:59:56Z")
You do not need a enterprise subscription for `reroute` -- it is part of the core free and open functionality. ---
**Author:** ![Namita\_Jaokar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/namita_jaokar/32/104106_2.png) [@Namita\_Jaokar](https://discuss.elastic.co/u/Namita_Jaokar)\ **Post date:** [November 22, 2023, 6:20am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/7 "2023-11-22T06:20:19Z")
Thanks for confirming @axw . Best Regards, Namita ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2023, 6:20am UTC](https://discuss.elastic.co/t/create-custom-data-streams-per-service/346710/8 "2023-12-20T06:20:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.