Elastic self manage / architecture

Elastic Stack Elasticsearch
1

Hello everyone !
i am very new to elastic and while going through the documentation i was not able to find somethings ,so if possible help me understand these and also provide the documentation link so i can verify (in prepective of SIEM)

q1. basic standalone & distributed architecture with main components and internal communication ports ? is their any refrence image of the architecture showing all components and ports (i have already seen hot/frozen architecture and it is not the basic one also it dosnt have the port numbers // also gone through ingest architecture in documentation /// componets i know are kibana -Elasticsearch-logstash-fleet-xpack if possible explain them and give me a basic diagram from documentation

q2. while reading about logstash and elastic agent i am not able to understand why are the major advantages of using logstash over the elastic agent or where the elastic agent should be used & where logstash (gone through ingest architecture documentation and got to know thoes 4 points like PQ ,proxy and etc are those only points but their also it is mentioned the used when we tranfer data from eelastic agent ,what if dont use elastic agent

q3. problem in understanding fleet & fleet server like ? first it seemed that fleet is on kibana but it is the web-ui only so if asked where is fleet as a component so the ans would be kibana or Elasticsearch & also the clarification /// dif between fleet server and standalone like why to use each or which one to prefer when

q4. what are the recommended os for slef mange like ubuntu or windows or mac (already gone through support matrix pdf ) but it dosnt state which are recommended it just tell which are supported which are not

q5. lets say i have to take logs from checkpoint firewall and i find it hard to write filter in logstash so i decide to use elastic agent and decided to use the checkpoint integration so my question is where should be elastic agent installed on seprate vm or windows or on the vm which is running Elasticsearch ? and in that case what would be the architecture like

2

Hello,

I don't think there are any diagrams for self-managed architectures as this depends entirely on the user requirements, it is up to each user to understand how the tools communicate with each other and to design the architecture for its use case.

The Elastic Stack documentation in the Elastic Fundamentals part is a good starting point.

Also, about the ports, you need to check the Self-Managed documentation on how to deploy it, it has this information, like this one.

There is not a single documentation, you will need to navigate to multiple documentations to understand how the stack works before deploying it in production.

But basically you store data in Elasticsearch and can use Kibana to visualize, make queries and create dashboards, Logstash and Elastic Agent are used to collect and ingest data into Elasticsearch, Logstash is used in more advanced use cases, Fleet is the managing interface for Elastic Agents, it is a page inside Kibana, but it is also an different component running in a different server.

It depends on your use case, both Logstash and Elastic Agent can be use to collect and receive data, Logstash provides you more flexiblity and some things are only possible with Logstash like enriching data making requests to external endpoints.

Logstash is more used in advanced use cases and you have to create your own configurations to parse the data, Elastic Agent has many native integrations with the parse for the data already done, this parse is executed in Elasticsearch using Ingest Pipelines, Elastic Agent per default will just get the data.

For most uses cases you can use just Elastic Agent.

Fleet server is used to centrally manage the deployment of your Elastic Agents, the fleet server is a service the runs on an Elastic Agent, normally you will have a dedicated machine to run this agent that will have the Fleet Server integration, the management is done through Kibana interface where you have a page to configure your policies and manage your agents.

As mentioned, you will need to check multiple documentations to understand better the tools, for Fleet you can check this one.

Standalone mode is when you manage the agent configuration files yourself, it is also considered an advanced case.

There are no recomendations, it is up to the user, personally I would use a Linux distribution, Ubuntu or Rocky Linux.

It is better to have a dedicated machine to run Elastic Agent to receive data from external sources.

Also, it is recommended that the hosts running Elasticsearch are dedicated to running Elasticsearch and do not run anything else.

3

thank you very much!! got the hang of much of the parts still had some doubt in logstash one

can you explain the enriching data more brifly with example (if i have to guess logstash can use some kind of lookups something like that)?

also explain “making requests to external endpoints“ with example ?(like see logstash can do this but elastic agent cant // and also state any of the advance usecase with example

so that i can have a very clear idea of these can be someuse cases where i should prefer logstash instead of elastic agent only

because in my mind we should use logstash when
1.we want to send data to mutiple outputs (which elastic agent cant do )
2.PQ it has presistent queue to handle temporary
3.if we dont have integration avaible for data source or we ourself want to write whole parser

rather than that we have processor in elastic agent or also in ingest pipeline which can help to achive more flexibility