# Can I have separate apm index log for each of my application **URL:** \ **Category:** APM\ **Tags:** server, ui\ **Created:** [April 25, 2023, 2:53am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692 "2023-04-25T02:53:04Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [April 25, 2023, 2:53am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/1 "2023-04-25T02:53:05Z")
I currently have 2 nodejs and 1 php app, and all of them are using the same apm server url and token to set up the apm agent. In Kibana discover, I found out there is a index pattern called "traces-apm\*,apm-_,logs-apm_,apm-_,mertrics-apm_,apm-\*" which contains all the transaction data for all of my app. I wonder if it is possible to have each of my application have its own index for stroing the log Currently I have 3 services in my APM nodejs-1 nodejs-2 php-1 I wonder if I can achieve something like that in my index pattern, nodejs1-apm-\* nodejs2-apm-\* php-1 -apm-\* Then I will have 3 index pattern respectively and each one of them is belongs to the related application base on the prefix. Thank you ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [April 26, 2023, 7:22am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/2 "2023-04-26T07:22:48Z")
Hi @JasonREC , in versions `>= 8.x` the apm server follows the general datastream naming template of the format `{data_stream.type}-{data_stream.dataset}-{data_stream.namespace}` to ingest data to Elasticsearch. It currently sends all trace events to a common index, similar for error logs and metrics (with the exception that app specific metrics are written to a dedicated app specific data stream). There is currently little flexibility in changing this when writing to ES directly from APM Server. But I suggest you take a look at how you can [use Kibana spaces to control access to apm data](https://www.elastic.co/guide/en/kibana/current/apm-spaces.html). It sounds like this might be exactly what you are looking for. ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [April 28, 2023, 3:15am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/3 "2023-04-28T03:15:44Z")
Thanks @simitt, I will have a look! ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 3, 2023, 1:44am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/4 "2023-05-03T01:44:40Z")
![php-enviroment_name](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfbbc7f4689bc9a2e79600fbce43c0b2cc89d198.png) Hi, @simitt , I have read the document, and here says I have to configure the php service enviorment setting to either staging or production so as to make the filter works. If I have only 1 instance machine which running 2 php app\*\*. one is for staging and another one is for produnction\*\*. How can I specify that one's data need to be sent with the service enviroment staging or production. Becuase there is only 1 php.ini file in one machine(correct me if I am wroing), if I specify service name is staging then I cannot specify that another app to send data with service eniroment.production. ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [May 8, 2023, 12:48pm UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/5 "2023-05-08T12:48:02Z")
Hi @JasonREC , my previous understanding was that you wanted to have different access control per `service.name` and not per environment. The `env` example in the docs I shared, is just meant as an example how you can leverage kibana spaces for more fine granular access control. I think you would want to achieve what has been shown for different environments, for your different services. How I understood your case was that you would want to 1. create filtered aliases with the service names in the alias (instead of the env) 2. create kibana spaces, one for every service 3. update the apm index pattern per space 4. create kibana access roles (again per service rather than env) 5. assign users to roles With that it should not be necessary to configure the environment settings in your php app. Let me know if I misunderstood you. ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 9, 2023, 1:30am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/6 "2023-05-09T01:30:03Z")
Hi @simitt, No, you're right. I followd the doc that you provided and based on the service name (instead of the env which used in the doc's example). and I found out when you create a space (step 2), and update the apm index(step3), other space index pattern will get overwrittern. let say I have the default space, now I create a space called php. if I fill out the "php" alias filiter on apm index pattern, default space am index pattern will also be showing the data regardings to "php" index pattern. I wonder if it is normal. Thank you! ---
**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\ **Post date:** [May 9, 2023, 6:10am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/7 "2023-05-09T06:10:49Z")
@JasonREC can you share screenshots for the APM UI index settings of both spaces, and the (JSON responses of the) aliases you have configured, including the alias filter? ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 9, 2023, 6:41am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/8 "2023-05-09T06:41:01Z")
Hi @dgieselaar Thanks for helping ![de-apm-ui](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1ce8c8a57e1a1616bc4b6395db33789acc6f4631.png) ![re-apm-ui](https://us1.discourse-cdn.com/elastic/original/3X/7/f/7fd2b9706014433bc59436992928a86faef56caf.png) ![traces](https://us1.discourse-cdn.com/elastic/original/3X/1/6/16b7fae0f3cb05f05fe9056864a81b6484aeb925.png) ![logs](https://us1.discourse-cdn.com/elastic/original/3X/4/3/4332e8d346967cf1a7ec7f018f2c1fcac0867038.png) ![metic](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac55fa2e1c622faa5520eb304e1b0f354ce7a59.png) When I set the apm indices in the "re" space, the default space will also automactilly be updated using "re" prefix. ---
**Author:** ![Rubyfaina](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@Rubyfaina](https://discuss.elastic.co/u/Rubyfaina)\ **Post date:** [May 9, 2023, 6:52am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/9 "2023-05-09T06:52:37Z")
> [@JasonREC](#): > > I currently have 2 nodejs and 1 php app, and all of them are using the same apm server url and token to set up the apm agent. > In Kibana discover, I found out there is a index pattern called "traces-apm\*,apm-_,logs-apm_,apm-_,mertrics-apm_,apm-\*" which contains all the transaction data for all of my app. > > I wonder if it is possible to have each of my application have its own index for stroing the log > > Currently I have 3 services in my APM > nodejs-1 > nodejs-2 > php-1 > > I wonder if I can achieve something like that in my index pattern, > nodejs1-apm-\* > nodejs2-apm-\* > php-1 -apm-\* > Then I will have 3 index pattern respectively and each one of them is belongs to the related application base on the prefix. > > Thank you Yes, you can have a separate index log for each of your applications. The index pattern can be created using the design you mentioned above. Change the APM server URL and token for each of your applications to ensure that data is routed to the appropriate index. This ensures that the log data is routed to the relevant index. ---
**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\ **Post date:** [May 9, 2023, 12:06pm UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/10 "2023-05-09T12:06:25Z")
@JasonREC to clarify, you have not made changes to the APM UI index settings for the default space, and if you update them for a different space, the changes are also applied to the default space? Specifically, you set `re-metrics-apm` in the second space, and instead of the field being empty in the default space as you would expect, it also shows `re-metrics-apm` there even though you have not made that change yourself? Maybe this question seems redundant but I can't reproduce it quickly. What version are you on? ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 10, 2023, 1:33am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/11 "2023-05-10T01:33:12Z")
> [@dgieselaar](#): > > , it also shows `re-metrics-apm` there even though you have not made that change yourself? Maybe this question seems redundant but I can't reproduce it quickly. What version are you on Hi, @dgieselaar Yes!! that's exactly what I encoutered. I think I am currenly on 8 ![elastic-version](https://us1.discourse-cdn.com/elastic/original/3X/4/8/482047ac004ea1a16b81a4433b1c1ee5a0ca3280.png) ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 10, 2023, 1:45am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/12 "2023-05-10T01:45:46Z")
Hi @Rubyfaina , I am not sure If I would like to have each app has its own cloud APM server, I have to pay for the increased APM server? ---
**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\ **Post date:** [May 10, 2023, 2:36pm UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/13 "2023-05-10T14:36:49Z")
@jasonREC are you able to upgrade to a more recent version? I cannot reproduce it in the latest versions. ---
**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\ **Post date:** [May 11, 2023, 2:31am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/14 "2023-05-11T02:31:28Z")
Hi, @dgieselaar now the bug is fixed after upgrade to 8.71 but it is a bit weird that I can only see **re-index-pattern** in default space, and I only see without any alias filter index pattern in "re" space. somehow they are inverted. ---
**Author:** ![dgieselaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dgieselaar/32/46947_2.png) [@dgieselaar](https://discuss.elastic.co/u/dgieselaar)\ **Post date:** [May 19, 2023, 7:34am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/15 "2023-05-19T07:34:27Z")
@JasonREC Sorry for the late reply. Can you clarify what you mean? Maybe share some screenshots? ---
**Author:** ![abhidwi27](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhidwi27/32/120097_2.png) [@abhidwi27](https://discuss.elastic.co/u/abhidwi27)\ **Post date:** [May 25, 2023, 2:35pm UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/16 "2023-05-25T14:35:19Z")
Hi @dgieselaar@JasonREC@simitt I have an identical requirement. I have set up the Elastic Stack version 8.7 in a self-managed manner. Here's my setup: I'm running Docker on an AWS EC2 instance. Inside the Docker container, I have a 3-node cluster and Kibana. To achieve this, I referred to the docker-compose.yml file mentioned in the Elastic 8.7 documentation. Additionally, I've configured a Fleet server. As part of this configuration, I enabled the "APM integration" feature on the Kibana dashboard, which I'll refer to as "dev-apm-integration." This feature spawns an APM server on the same host, listening on port 8200. In my Kubernetes cluster, I have multiple pods running different services. Let's consider two microservices running as pods: ```auto my-service-1 ``` ```auto my-service-2 ``` To start both services, I use the following command: ```auto java -javaagent:elastic-apm-agent-1.38.0.jar -Delastic.apm.server_url=https://:8200 -Delastic.apm.secret_token= -Delastic.apm.application_packages= -Delastic.apm.service_name=abhi-service-content -Delastic.apm.environment= my-service-1.jar ``` ```auto java -javaagent:elastic-apm-agent-1.38.0.jar -Delastic.apm.server_url=https://:8200 -Delastic.apm.secret_token= -Delastic.apm.application_packages= -Delastic.apm.service_name=abhi-service-content -Delastic.apm.environment= my-service-2.jar ``` Both services push data to the same APM server that is connected to the Fleet server. Even if I create separate spaces, such as "my-service-1-space" and "my-service-2-space," and modify the above commands (change elastic.apm.environment) accordingly, the data still gets sent to the same apm server and ultimately ends up in the same indices. What I desire is to have different data streams for each service so that I can apply different ILM (Index Lifecycle Management) policies to the data. Here's how I expect the configuration to work: ```auto For my-service-1: Data should go to log-my-service-1-apm, trace-my-service-1-apm, and metric-my-service-1-apm. ``` ```auto For my-service-2: Data should go to log-my-service-2-apm, trace-my-service-2-apm, and metric-my-service-2-apm. ``` I would greatly appreciate any guidance on how to achieve this configuration. Thank you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 15, 2023, 10:35am UTC](https://discuss.elastic.co/t/can-i-have-separate-apm-index-log-for-each-of-my-application/330692/17 "2023-06-15T10:35:39Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.