Hello
I want to try out the integration feature but Im kind of lost.
Im gonna have my ELK stack all on one server. The reason I mention this is because there is a Fleek/Elastic Agent being talked about and Im not sure what that does.
I only want to use Elastic, Logstash and Kibana. Nothing else, unless it is required.
Could you please give me a ELI5 ?
I wanted to start with the Fortinet Fortigate one to start and then move on.
Nothing?
Hello,
What is exactly your question? Did you check any documention?
The Elastic Agent is the now recommended tool to collect logs and receive data when using Elasticsearch, it will run integrations, which are preconfigured collectors and parsers, the logs are parsed using ingest pipelines in Elasticsearch, each integration will use one or more ingest pipelines to parse the logs into ecs fields.
Fleet Server is the tool that will manage the elastic agent. Each agent is part of one agent policy, and the Fleet server is responsable into distributing those policies to the agents, the policies specifies what the agent will collect and where it will send the data.
Integrations requires and Elastic Agent to receive/collect the data.
If you want to use just Logstash for that you will need to parse the logs yourself using Logstash filters, you can also configure it to use an ingest pipeline in Elasticsearch, but this is a little more advanced and I would not recommend for someone that is just starting with the stack.
What is exactly your question? Did you check any documention?
I believe the documentation might have further confused me, so I apoligize before hand.
From what I understand, the Elastic Agent is simply a component that runs on a server BUT the documentation says only ONE can run on a server. So, if I run it on my ELK server, I can only receive logs from one place. Correct?
The Elastic Agent is the now recommended tool to collect logs and receive data when using Elasticsearch, it will run integrations, which are preconfigured collectors and parsers, the logs are parsed using ingest pipelines in Elasticsearch, each integration will use one or more ingest pipelines to parse the logs into ecs fields.
So I need to run the Elastic Agent and then in Kibana configure the integration? Thats the part that maybe isnt clear to me.
Fleet Server is the tool that will manage the elastic agent. Each agent is part of one agent policy, and the Fleet server is responsable into distributing those policies to the agents, the policies specifies what the agent will collect and where it will send the data.
Why do I need more agents? Thats another confusion.
Integrations requires and Elastic Agent to receive/collect the data.
OK, so I NEED the Elastic Agent. Got it.
If you want to use just Logstash for that you will need to parse the logs yourself using Logstash filters, you can also configure it to use an ingest pipeline in Elasticsearch, but this is a little more advanced and I would not recommend for someone that is just starting with the stack.
Yeah, Ive set up Logstash pipelines but I rather go another cleaner way.
Thank you for your comments and help
No, each agent can run multiple integrations, so you can receive logs from multiple places in the same agents.
This documentation has an overview on how it works and the multiple components of it.
You use the Fleet UI in Kibana to create an agent policy and add the integrations you want, the Fleet Server will then deploy this configuration into your agents.
This depends on what you need to collect and what your infrastructure looks like, for example if you have multiple servers and want to collect log files from those servers, than each one will need an agent, or if you want to collect logs from s3 buckets and you have a lot of lots, you may need multiple agents to balance the load, same thing with network devices, you may need to use multiple agents to balance the load.
No, each agent can run multiple integrations, so you can receive logs from multiple places in the same agents.
This [documentation](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html) has an overview on how it works and the multiple components of it.
The how to and documentation isnt clear but if I can use only one agent on the same server....
You use the Fleet UI in Kibana to create an agent policy and add the integrations you want, the Fleet Server will then deploy this configuration into your agents.
Thats something I also dont get about the agent policy; I feel like it should be able to set a ILM for setting hot, warm and cold phases but I dont see it. It isnt intuitive
This depends on what you need to collect and what your infrastructure looks like, for example if you have multiple servers and want to collect log files from those servers, than each one will need an agent
I dont understand this, Im sorry. If I have various Windows servers, why do I need various agents? How does that work?
same thing with network devices, you may need to use multiple agents to balance the load.
Im going to only capture Windows servers, Linux servers, network devices, and thats it. How many agents do I need? Lets keep it simple 
What is not clear? You can only run one agent per server, this is mentioned in multiple places.
So, 1 agent per serer, multiple integrations per agent.
The ILM policy for the agent and its integrations is managed by the Fleet Server, all integrations will use the same ILM policy, but this can be customized.
Well, if you want to collect logs from the server, you need an agent running on the server to get those logs and send to elasticsearch.
The Agent is a log collector, to get the logs it needs to have access to the logs, so you need to have one agent per server.
As mentioned, this depends entirely on your infrastructure. If you want to get logs from a server, you need an agent per server, so if you have 100 servers, you need 100 agents, one in each server.
For network devices is a little different, because you need an agent listening on some port, so the best approach is to spin-up some servers that will receive the logs from your network device, like they were Syslog servers for example.
How many of those servers you need depends on the volume of your logs.
The best way to see how all this work is to do a proof of concept to learn and see what you will need in your specific use case, a common mistake that I constantly see is people trying to build everything already in production without testing and learning how it works.
What is not clear? You can only run one agent per server, this is mentioned in multiple places.
So, 1 agent per serer, multiple integrations per agent.
So what I understand from what you are saying is that I install the Elastic Agent THE SAME PLACE where I have my ELK stack installed and done? To collect logs from the server itself???
The ILM policy for the agent and its integrations is managed by the Fleet Server, **all** integrations will use the same ILM policy, but this can be customized.
The Fleet Server is required when there are SEVERAL servers with SEVERAL Elastic Agent installations. Else, do I really need a Fleet Server? Which, again, in this case, it would be installed in the same server as the ELK Stack.....
Well, if you want to collect logs from the server, you need an agent running on the server to get those logs and send to elasticsearch.
The Agent is a log collector, to get the logs it needs to have access to the logs, so you need to have one agent per server.
Ah, so I need to install Elastic Agent on ALSO the Windows and Linux Servers?
As mentioned, this depends entirely on your infrastructure.
Windows Servers
Linux Servers
Network devices (firewall, switches, etc.)
For example, if you have 3 servers running Elasticsearch and you want to get the logs from those servers, then you need to install the Elastic Agent in each one of the servers.
The Elastic Agent is a log collector, if you want to get a log from a server, you need to agent running on that server. You have thousands of servers and want to get logs from all of them? Then you need to install the agent on each one of them.
The Fleet Server is requires no matter if you have one agent or thousand of agents, the management of the agents is done by the fleet server. In the documentation linked before the are links to other documentations that explain how the agent works, this one here explains what is a Fleet Server, what it will do and the deployment models.
This is description of a Fleet Server:
Fleet Server is a component that connects Elastic Agents to Fleet. It supports many Elastic Agent connections and serves as a control plane for updating agent policies, collecting status information, and coordinating actions across Elastic Agents. It also provides a scalable architecture. As the size of your agent deployment grows, you can deploy additional Fleet Servers to manage the increased workload.
Where you will install the fleet server is up to you, I recommend having it on a separate server.
Yes, if you want to get the logs from the server you need an agent installed on the server to be able to get the logs.
For example, if you have 3 servers running Elasticsearch and you want to get the logs from those servers, then you need to install the Elastic Agent in each one of the servers.
I think thats why Im not explaining correctly, sorry
I will only have ONE server and that server is going to contain everything: Elasticsearch, Kibana, Logstash.....Everything.....
The Elastic Agent is a log collector, if you want to get a log from a server, you need to agent running on that server. You have thousands of servers and want to get logs from all of them? Then you need to install the agent on each one of them.
OK, I got you I think. I need to install Elastic Agent on all of the servers I want to collect logs FROM.
WINDOWSSERVER2022-01
WINDOWSSERVER2022-02
WINDOWSSERVER2022-03
WINDOWSSERVER2022-04
LINUXSERVER-01
LINUXSERVER-02
All of these need to have it (Elastic Agent) installed to send logs to ElasticKibanaLogstashServer-01, right? I think I understand now, thank you
The Fleet Server is requires no matter if you have one agent or thousand of agents, the management of the agents is done by the fleet server. In the documentation linked before the are links to other documentations that explain how the agent works, this one [here](https://www.elastic.co/guide/en/fleet/current/fleet-server.html) explains what is a Fleet Server, what it will do and the deployment models.
So it is required?
Where you will install the fleet server is up to you, I recommend having it on a separate server.
Thats not possible. Is it possible to have it installed on the same server as Elastic, Kibana and Logstash?
Yes, if you want to get the logs from the server you need an agent installed on the server to be able to get the logs.
OK, so my first Elastic Agent will be installed on the Elastic, Kibana, Logstash server itself. Gotcha
Yeah, you could have everything on the same server, but this is not recommended for many reasons, but if you want you can, just keep in mind that the performance can be impacted.
Also, if you are going to use Elastic Agent, maybe you do not need Logstash.
Yes, you will need one Agent on each onf of those servers.
Deploying Fleet managed agents is the recommended way to deploy agents, and in this deployment model a Fleet Server is required, however it is possible to deploy agents in a standalone mode where you manage everything, this is an advanced use case that assumes that you already has experience with both Elastic Stack and Elastic Agents, I would not recommend anyone starting with the Elastic stack to do this.
The differences are explained in the documentation linked in the previous post.
Yes, you can have everything in one server, it is not recommended, and the performance may not be good, but you can.
Yeah, you could have everything on the same server, but this is not recommended for many reasons, but if you want you can, just keep in mind that the performance can be impacted.
Also, if you are going to use Elastic Agent, maybe you do not need Logstash.
Due to the infrastructure, everything is gonna be on the same server.
I have Logstash in case I need additional logs and parse them.
Yes, you will need one Agent on each onf of those servers.
OK, perfect. I IMAGINE the agents installed on the Windows computers collect logs and then send them directly to Elasticsearch right?
Deploying Fleet managed agents is the recommended way to deploy agents, and in this deployment model a Fleet Server is required, however it is possible to deploy agents in a standalone mode where you manage everything, this is an advanced use case that assumes that you already has experience with both Elastic Stack and Elastic Agents, I would not recommend anyone starting with the Elastic stack to do this.
The differences are explained in the documentation linked in the previous post.
Yes, the Fleet way is the suggested way....that being said, the documentation is very badly worded on what is what, how is how, and why is why.
Yes, you can have everything in one server, it is not recommended, and the performance may not be good, but you can.
Yeah, I accept that.
I installed the Fleet and installed the Agent BUT the integration with Fortinet is not working correctly.
A traffic capture SEEMS to show the traffic arriving to ELK but it isnt processed in any way, shape or form.
Do you need any other information?
You configuration is wrong, you are using localhost, you should use 0.0.0.0.
Using localhost means that the input will only listen on 127.0.0.1, which means that it will only accept requests from the same host.
I thought that was the interface I was gonna listen to which could be 127.0.0.1, the actual IP or 0.0.0.0 which is all interfaces.
I changed it to 0.0.0.0 but same thing
Maybe Im missing something obvious, so could someone please state a step by step on how to get a integration working?
Things I have done:
Elastic Agent is installed and working:
As you saw, logs are incoming from the Forti:
The integration is setup:
Am I missing anything?
Any suggestions? Thank you
Check the Elastic Agent logs in the server where the integration is running to see if there is any error.
Also, can you do a tcp dump on the same server on the port you are sending the data and share it as plain text? Use the preformatted text option, the </> button, it is not possible to see the wireshark print you share, just got a quick tcp dump and share it.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
