I've used Elastic SIEM before but always in environments where the Elastic agent can be deployed to endpoints and servers. In my current environment I cannot deploy the agent. I have both Linux and Windows boxes.
Questions
Is there a way to get the logs into Elastic without agents on every log source?
Can I setup fleet or an agent on a linux/Windows box and use it as a forwarder to collect logs from different sources?
I know this is probably possible with additional development/scripting but I would like to know if there is a way to have this done without such development effort i.e. with minimal configuration out of the box.
In many other SIEMs, you simply "add" log sources by pointing rsyslog to the SIEM server for Linux/Network devices or Windows log sources by specifying user/password within the SIEM and it uses MSRPC to login to Windows servers and collect the logs. This is true for most of the major SIEM players. Without something like this I think Elastic as a SIEM will be very limiting for most companies since it brings yet another agent that needs to be installed or require development effort that lean teams may not possess.
It depends on which kind of logs you want, for linux system for example, if you want jus the system logs, the ones sent to /var/log/messages or /var/log/syslog, you can use rsyslog to redirect the logs to another server where you will install an Elastic Agent to get those logs.
If you want application logs like NGINX for example, you would need to make sure that it is logging to rsyslog and then redirect these logs, but you would probably need to make some adjusts on Elasticsearch with a custom inget pipeline so these logs can be parsed by the correct pipeline.
In resume, in Linux, if you can log to rsyslog you can redirect it to another server running Elastic Agent.
For windows you can configure your windows hosts to forward the Events to a server where you would then install Elastic Agent.
The level of log collection possible without agents depends on your specific monitoring requirements:
For deep system monitoring (like process executions, file changes, and network connections), an agent running directly on the system is necessary. However, for less intensive monitoring needs, there are several agentless options:
System Logs:
These can be collected without agents using rsyslog
rsyslog can be configured to write directly to Elasticsearch using the omelasticsearch plugin
This provides a lightweight solution for basic system monitoring
Application Logs:
Can be redirected to write through rsyslog
This allows centralized collection without requiring agents on each system
For centralized collection, you have two main options:
Using Elastic Agent as a central collector:
Deploy on a single server to receive logs from multiple sources
Provides built-in parsing and processing capabilities
Using Logstash as a central collection point:
Ideal when you need data transformation or enrichment
Offers powerful processing capabilities before forwarding to Elasticsearch
The choice between these approaches depends on your specific requirements for data processing and transformation.
Thank you @Nick_Alayil@leandrojmp for the guidance. Sorry it took this long to respond as I was away from office.
I succeeded to get the logs into Elastic from linux rsyslog using fleet and the "custom tcp log" plugin. I.e. the policy is applied on my fleet server. I will try with the omelasticsearch plugin as well.
For windows I installed the agent on an Event collector server and I'm forwarding logs to the collector. The only issue is that the collector receives logs in the "Forwarded events" channel which the agent has no way of collecting AFAIK. When I edit the system integration, I can only collect Application, Security, and System. Any further ideas what I can do here?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.