Can I have separate apm index log for each of my application

Elastic Observability APM
,
1

I currently have 2 nodejs and 1 php app, and all of them are using the same apm server url and token to set up the apm agent.
In Kibana discover, I found out there is a index pattern called "traces-apm*,apm-,logs-apm,apm-,mertrics-apm,apm-*" which contains all the transaction data for all of my app.

I wonder if it is possible to have each of my application have its own index for stroing the log

Currently I have 3 services in my APM
nodejs-1
nodejs-2
php-1

I wonder if I can achieve something like that in my index pattern,
nodejs1-apm-*
nodejs2-apm-*
php-1 -apm-*
Then I will have 3 index pattern respectively and each one of them is belongs to the related application base on the prefix.

Thank you

2

Hi @JasonREC ,
in versions >= 8.x the apm server follows the general datastream naming template of the format {data_stream.type}-{data_stream.dataset}-{data_stream.namespace} to ingest data to Elasticsearch. It currently sends all trace events to a common index, similar for error logs and metrics (with the exception that app specific metrics are written to a dedicated app specific data stream). There is currently little flexibility in changing this when writing to ES directly from APM Server.

But I suggest you take a look at how you can use Kibana spaces to control access to apm data. It sounds like this might be exactly what you are looking for.

3

Thanks @simitt, I will have a look!

4

php-enviroment_name
php-enviroment_name981×366 43.3 KB

Hi, @simitt , I have read the document, and here says I have to configure the php service enviorment setting to either staging or production so as to make the filter works.

If I have only 1 instance machine which running 2 php app**. one is for staging and another one is for produnction**. How can I specify that one's data need to be sent with the service enviroment staging or production. Becuase there is only 1 php.ini file in one machine(correct me if I am wroing), if I specify service name is staging then I cannot specify that another app to send data with service eniroment.production.

5

Hi @JasonREC ,
my previous understanding was that you wanted to have different access control per service.name and not per environment.
The env example in the docs I shared, is just meant as an example how you can leverage kibana spaces for more fine granular access control. I think you would want to achieve what has been shown for different environments, for your different services.
How I understood your case was that you would want to

  1. create filtered aliases with the service names in the alias (instead of the env)
  2. create kibana spaces, one for every service
  3. update the apm index pattern per space
  4. create kibana access roles (again per service rather than env)
  5. assign users to roles

With that it should not be necessary to configure the environment settings in your php app. Let me know if I misunderstood you.

6

Hi @simitt,

No, you're right. I followd the doc that you provided and based on the service name (instead of the env which used in the doc's example). and I found out when you create a space (step 2), and update the apm index(step3), other space index pattern will get overwrittern.

let say I have the default space, now I create a space called php.
if I fill out the "php" alias filiter on apm index pattern, default space am index pattern will also be showing the data regardings to "php" index pattern. I wonder if it is normal.

Thank you!

7

@JasonREC can you share screenshots for the APM UI index settings of both spaces, and the (JSON responses of the) aliases you have configured, including the alias filter?

8

Hi @dgieselaar

Thanks for helping

de-apm-ui
de-apm-ui1766×915 92 KB

re-apm-ui
re-apm-ui1772×963 95 KB

traces
traces1895×839 32.6 KB

logs
logs1873×778 29.6 KB

metic
metic1903×649 30.8 KB

When I set the apm indices in the "re" space, the default space will also automactilly be updated using "re" prefix.

9

Yes, you can have a separate index log for each of your applications. The index pattern can be created using the design you mentioned above. Change the APM server URL and token for each of your applications to ensure that data is routed to the appropriate index. This ensures that the log data is routed to the relevant index.

10

@JasonREC to clarify, you have not made changes to the APM UI index settings for the default space, and if you update them for a different space, the changes are also applied to the default space? Specifically, you set re-metrics-apm in the second space, and instead of the field being empty in the default space as you would expect, it also shows re-metrics-apm there even though you have not made that change yourself? Maybe this question seems redundant but I can't reproduce it quickly. What version are you on?

1 Like
11

Hi, @dgieselaar
Yes!! that's exactly what I encoutered.
I think I am currenly on 8

elastic-version
elastic-version1846×938 73.7 KB

12

Hi @Rubyfaina ,

I am not sure If I would like to have each app has its own cloud APM server, I have to pay for the increased APM server?

13

@jasonREC are you able to upgrade to a more recent version? I cannot reproduce it in the latest versions.

14

Hi, @dgieselaar

now the bug is fixed after upgrade to 8.71

but it is a bit weird that I can only see re-index-pattern in default space, and I only see without any alias filter index pattern in "re" space. somehow they are inverted.

15

@JasonREC Sorry for the late reply. Can you clarify what you mean? Maybe share some screenshots?

16

Hi @dgieselaar @JasonREC @simitt

I have an identical requirement.

I have set up the Elastic Stack version 8.7 in a self-managed manner. Here's my setup:

I'm running Docker on an AWS EC2 instance. Inside the Docker container, I have a 3-node cluster and Kibana. To achieve this, I referred to the docker-compose.yml file mentioned in the Elastic 8.7 documentation. Additionally, I've configured a Fleet server. As part of this configuration, I enabled the "APM integration" feature on the Kibana dashboard, which I'll refer to as "dev-apm-integration." This feature spawns an APM server on the same host, listening on port 8200.

In my Kubernetes cluster, I have multiple pods running different services. Let's consider two microservices running as pods:

my-service-1

my-service-2

To start both services, I use the following command:

java -javaagent:elastic-apm-agent-1.38.0.jar -Delastic.apm.server_url=https://<apm-server-host>:8200 -Delastic.apm.secret_token=<mytoken> -Delastic.apm.application_packages=<my-packge> -Delastic.apm.service_name=abhi-service-content -Delastic.apm.environment=<any-env-name> my-service-1.jar

java -javaagent:elastic-apm-agent-1.38.0.jar -Delastic.apm.server_url=https://<apm-server-host>:8200 -Delastic.apm.secret_token=<mytoken> -Delastic.apm.application_packages=<my-packge> -Delastic.apm.service_name=abhi-service-content -Delastic.apm.environment=<any-env-name> my-service-2.jar

Both services push data to the same APM server that is connected to the Fleet server. Even if I create separate spaces, such as "my-service-1-space" and "my-service-2-space," and modify the above commands (change elastic.apm.environment) accordingly, the data still gets sent to the same apm server and ultimately ends up in the same indices.

What I desire is to have different data streams for each service so that I can apply different ILM (Index Lifecycle Management) policies to the data. Here's how I expect the configuration to work:

For my-service-1: Data should go to log-my-service-1-apm, trace-my-service-1-apm, and metric-my-service-1-apm.

For my-service-2: Data should go to log-my-service-2-apm, trace-my-service-2-apm, and metric-my-service-2-apm.

I would greatly appreciate any guidance on how to achieve this configuration. Thank you.

17

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.