Hello friends, I'm a bit confused, I have 2 agent policies and 2 APM integrations, the first agent policy is by default and this one has 2 integrations "Elastic APM" and "Fleet-server" and another policy that I created with 1 elastic agent associated and 3 integrations, one of them is another APM integration that I customized. So it happens that I instrument an application with the APM agent but it doesn't work with the policy that I customize, but with the default policy that Elastic created.
Sorry, there is not enough info in order to provide any useful feedback.
another APM integration that I customized
How did you customize? What did you change?
but it doesn't work with the policy that I customize
What doesn't work? Did you install an APM agent and found errors in the agent logs? Is the Elastic agent emitting errors? Do you know whether the APM Server got installed and started by the Elastic Agent?
Hi @Eyal_Koren
I have 2 integration APM in differents Agen Policies, and maybe we want to have more Integrations APM to segment the data by namespace so
What is the best way to have APM integrations separated by namespace?
If I have 2 integratios APM, in the configurations I will set differents urls ? for example attached 2 pictures.
I have 2 APM Integrations so the same URL and HOST, but different namespace so if I have 2 JAVA applications in differents hosts and I want to send data
APP01 and HOST01 => first APM Integrations with namespace 104666
APP02 and HOST02 => Second APM Integrations with namespace 104555.
So:
I can put any domain in the URLs but should they be different in each Integration APM?
My URL to APM server in First APM Integration is https://776eb7da8f034a44b96571f88f2f237f.apm.us-east-1.aws.found.io:443 the second URL in the second APM integratios should be different?
Can I use the same host and port in each APM Integrations? 0.0.0.0:8200 in both APM Integrations?
How can I make this difference to segment APM by namespace? I will have to create one agent policy for each APM integration?
Currently, I have a policy (Elastic Policy BO CAVALI with 1 Agent (Host with elastic Agent) where is the APP Java and APM agent, but APM is write data using other policy Elastic Cloud Agent policy (this policy was create by default with fleet server Integration.
Sorry in advance if I got it all wrong, but as far as I understand there is a misconception here, so my answer is in accordance:
When you use Elastic APM, you can either use a hosted APM Server through Elastic cloud, or a self-managed APM Server (I refer to Fleet-managed as self-managed in this case). When you add the APM integration to your policy through Fleet, you basically choose to use a non-hosted APM Server. When an Elastic Agent receives a policy that contains the APM integration, it will start an APM Server on the machine it is installed at. Typically, this can be used locally alongside the monitored application, to which you would attach an APM agent, but you can also use Elastic Agent and APM integration to install and manage an APM Server that will be publicly available for APM agents on other hosts (in which case you will need to properly configure networking setups to allow that).
If I am not mistaken, the https://776eb7da8f.... URL you configured in your policy refers to a hosted APM Server or Elastic cloud. This doesn't make sense if you add an APM integration to a policy that you apply to an Elastic Agent on your managed host, as such policy basically configures another APM Server on the edge host.
Back to your request to use different namespaces: I am not really familiar with the topic, but I believe an APM Server sends data to a single namespace, so if you want multiple namespaces, you will have to have multiple APM Servers. Therefore, if you choose to use hosted APM Servers through your Elastic Cloud deployment, you will need to use multiple instances and configure them accordingly (I don't know how).
Another option that seems as a better fit to what you describe is to use a local APM Server on each host where you have an application you want to monitor with APM. Then, the host:port can be localhost:8200 and the URL can be http://localhost:8200. Accordingly, the server_url configuration for the APM Agent on that host can be omitted (as the default is http://localhost:8200).
The second option would work best if you only run one Java application on each host. If you run multiple Java applications on the same host, they will have to use the same APM Server, thus write to the same namespace.
Assuming that you have 2 Java applications and you want each to store data on a different namespace, you would create a policy for each application, install and enroll an Elastic Agent on each host that run those and apply each policy to the corresponding agent/s.
I hope this helps.
@Eyal_Koren good morning thanks for you answer. So, I understand very bad the concept of APM Server in Elastic Cloud(this options we want to use) and APM SErver self-managed.
We add 1 node APM SErver integration this server have APM and Fleet and this is FREE
because we want tos tar using APM, so I have a host with 2 APP JAVA. What is the way to use APM Server in Elastic CLoud correctly?
we only want agent APM in our host onpremise and the servers send application data to the elastic cloud.
APM Server is the component that prepares the APM data coming from APM Agents and sends it to Elasticsearch. Whether it is on cloud or on your hosts is not much of a difference, as eventually data needs to be sent from your hosts to your Elastic cloud deployment for storage. If you use APM Server on cloud - your hosts will be transferring raw APM traces and if you use APM on your edge machines - your hosts will be transferring processed APM traces.
Now it depends on what you need. If you specifically need data separation through namespaces, then I think local APM Server on the application hosts (through Fleet and Elastic Agent) is the choice that will give you the best flexibility.
However, make sure that namespaces are really what you need. You should be aware that your APM data is sliced and searched within the Kibana APM app based on fields, primarily service.name (configurable through the APM Agent service_name config) and environment (configurable through the environment config). If this kind of data-separation is good enough for you, then you can use a single APM Server to ingest data from as many applications/services (each instance with its own APM Agent) as you want, as long as it can handle the load. Therefore, in that case using the hosted cloud APM Server should be an easy way to go.
I hope this helps.
1 Like
@Eyal_Koren I have a question friend,
Here is the IP of the server that contains the APP registered? or is it the host or IP of the APM server, in this case the same as the URL of my server that Elastic gives me for free 1GB, because I have 2 agent policy with one integration APM each so when I send the data from the Java Agents, always send data to Default Agent Policy (Elastic created this in fleet) and My policies don't work.
something I'm doing bad Y_Y
Sorry, I don't understand, if you share your exactly policy, then maybe we can see what the issue is.
If you want an APM Server installed on the host where the Java app runs, your APM integration in the policy should be:
Host: localhost:8200
URL: http://localhost:8200
I am not sure which version you are using, but with the latest it looks something like this when you edit your policy and find the APM integration:
You should click the circled button, which will bring you to this configuration page:
I hope this helps.
Hi @Eyal_Koren sorry, my English is bad haha, but No, I want to use the APM Server Free 1GB from Elastic give us, only that I have 2 hosts with Elastic Agent installed, so I wanto to use only APM Server from Ealstic to send data in diferent datastream, so I configured 2 APM Integration, 1 in each Agen policy, but there is one Agen policy that Elastic created by default and APM is writing using this Agent policy and not my Agents policies, this is my problem, but all APM integrations is pointing to APM Server Elastic Free.
This is my Agent policies, I created only one and the other was created by default.
This is APM Integration from Agent policy default:
This Apm integrations is I created:
I pointing to url from my APM SErver Elastic CLoud and configured the namespace to my datastream, but when I send data with the parameters from application java in the host, this send data using the Agent policy by default and not that I created, Why?
An elastic-agent policy itself is only a configuration, it doesn't run or spin up any binaries. For a policy to take effect, you need to enroll an Elastic Agent into this policy.
For the Elastic Cloud setup, we create everything by default. This is the elastic-agent-on-cloud policy that you already mentioned, which is applied by an Elastic Agent running an APM Server on Elastic cloud.
It is not possible to enroll one Elastic Agent into multiple agent policies. So what you are trying to do, to apply a second policy to the same Elastic Agent, will not work.
Unfortunately dynamic namespaces are not yet supported. So if you need to write to different namespaces, you will either need to host your own Elastic Agent with a dedicated agent policy, or create a second Elastic cloud deployment and configure the other namespace there.
This overview over Elastic Agent explains the concepts in more details.
1 Like
Hi, @simitt thanks for your answer!
I don't understand "you will either need to host your own Elastic Agent with a dedicated agent policy".
With the integration server provided by Elastic Cloud for free, can I only use one APM integration in one Agent policy? and could only use the one that Elastic cloud gives by default?
Yes that's right. The Elastic Cloud Integrations Server runs one Elastic Agent with one pre-configured APM integration. You cannot add another APM Integration to this setup. You can change some of the APM configuration settings of this policy, such as the namespace. But unfortunately dynamic namespaces, or multiple namespaces for one APM Integration are not supported.
However, instead of writing data to different namespaces, maybe seperating the data in the APM UI view might be an option for you? Please check out how to use APM spaces in the Kibana UI for separating data. Let us know if this would be a valid option for your use case.
1 Like
Hi @simitt, just to check, if I want to create another APM integration we have to build another Elastic Cloud Integrations Server node in the cloud, but I can also implement an Onpremise APM Server, right?
The default policy that the Elastic Cloud integration server creates is somewhat restrictive, because it doesn't let me add other integrations or other hosts with elastic agents:
- Does this happen because it is the 1GB free version or every time I want to add another Integration Server type node, will it limit me to only those two integrations by default (APM and Fleet Server)?
- To send data via APM to elastic cloud with fleet from a host, is it mandatory to have Elastic Agent installed on the host?
if I want to create another APM integration we have to build another Elastic Cloud Integrations Server node in the cloud, but I can also implement an Onpremise APM Server, right?
Yes, either you run another one in Elastic cloud or you run an Elastic Agent with an apm integration yourself.
The default policy that the Elastic Cloud integration server creates is somewhat restrictive, because it doesn't let me add other integrations or other hosts with elastic agents:
If you added another integration to the Elastic cloud integration server, then this integration would run directly in cloud. This is not supported at this point.
You can self-host an Elastic Agent, enrolled to an elastic agent policy with any kinds of integrations that you'd like.
To send data via APM to elastic cloud with fleet from a host, is it mandatory to have Elastic Agent installed on the host?
No. The architecture is to run an Elastic APM agent per service from which you want to collect APM data. You then configure these APM agents to send data to the APM Server URL (which is the one from the Elastic Cloud hosted APM Integrations Server).
There is some naming overloading, but essentially Elastic APM agents != Elastic Agent. This overview hopefully makes things a bit more clear.
1 Like
@simitt so what is the recomendation if we want to have multiple APM Integrations because we need to separete datastreams and each DS has a different ILM per customer or area that needs monitor its applications ?
If I have a lot of customers in the future, I should have 1 apm server (cloud or self-managed) per client?
What depends on whether my APM server has more or less RAM?
Is it expensive to add an APM Server Cloud per customer?
If using spaces in the Kibana UI is not a valid option for you, as you'd like to have different retention policies per service, then yes you could decide to have a dedicated APM Server per use case.
Please refer to the prices shown at cloud.elastic.co for finding answers about costs.
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.