I want to explore Elastic APM to be used in some of our IT applications. The plan is to go with the free version of the self-managed Elastic APM. I learned four components of the Elastic APM solution - APM agents, APM Server/APM integration, Elasticsearch & Kibana. APM Agent needs to be installed in the servers which need to be under APM monitoring. However, rest three components need to be installed in separate servers/VM. I saw the installation guide to install these components which should be ok to start. I tried but was still unable to find some relevant information to prepare a complete plan to begin. It would be helpful if I get some references for the following information.
Does it require three separate servers/VM to install the components - APM Server/APM integration, Elasticsearch & Kibana? or a single instance is ok.
How can I get the infrastructure (CPU, Memory, storage) requirement for these 3 components?
How to calculate bandwidth requirements?
What is the impact on the server utilization after APM Agent installation? e.g, CPU & RAM usage.
To all respected members,
If anyone already did small-scale self-managed Elastic APM, it would be a great help if you can share dimensioning, deployment architecture, learning, and suggestion.
First of all a big complement for you to want to learn how to set everything up and doing it thoughtfully. All these answers are assuming a 1 node elastic cluster.
You could use a single instance but I'd recommend doing two. One instance for the APM integration (read Elastic Agent), and the other for Elasticsearch and Kibana.
It really depends how much data you are going to be sending and saving, but elastic says to try 4GB when running locally in docker. Personally, I'd start at 8G. Sounds like you are using VMs so adding more space shouldn't be a big deal, but start at 500G. CPU should be fine at normal defaults.
If it is just APM it's going to look like: c * a * o = bandwidth, where c is the number of apm clients, a is the amount of data each client sends in a given time period, and o is a constant to factor in the network communication overhead. In reality though starting small and adding one or two apm clients at a time will give you the concreate numbers you can use to project bandwidth (and infrastructure resources).
From my understanding the apm agents are going to talk to Elastic Agent (either through fleet or installed as a stand alone program). The elastic agent will then send all the data on to the elastic cluster from there. Therefore adding more apm agents should just cause a slight increase in resources at the elastic agent host. Speaking of that here's what I recommend:
VM instance 1 (elastic & kibana), 8GB memory, 500GB storage.
VM instance 2 (Elastic agent), 8GB memory, 50GB storage. Probably can get away with less storage.
I think you'll need to decide if you are going to install elastic agent with fleet, or as a standalone install.
As your needs grow you can add dedicated master nodes, additional data nodes, split out kibana to it's own vm instances, etc.
I'm trying to manage 2 VM as per your suggestion. Please let me know if there is any issue with using RHEL 8.
As the standalone elastic agent is for the advanced user, I want to go with Fleet-managed Elastic Agents. I've gone through some documentation to start installing the components; I'll start following all those after making the VMs ready. However, if there is any consolidated documentation with sequential steps to install and make the whole APM solution ready, please share.
As long as you are the 8.x version you should be fine with RHEL 8. Here's the support matrix for the nitty gritty. These elastic docs about on premises fleet install should help you out with the install if I'm understanding your requirements right. One final bit concerning RHEL 8 is you might run into issue with SELinux making it harder to configure services. This has nothing to do with elastic, but is still a hurdle that sometimes pops up. If you do run into issues with SELinux I highly recommend this post from RedHat. Good luck.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.