Can I filter a separate set of logs through logstash rather than straight to Elasticsearch?

Elastic Stack Logstash
1

Hello,

I currently have Elasticsearch and Kibana installed and configured. I installed fleet-server to manage my elastic-agents and installed elastic-agents to all my hosts and everything is working as should. My elastic-agents are forwarding logs from /var/log/secure, /var/log/messages, and /var/log/audit/audit.log from all my host straight to Elasticsearch.

I do have one particular log located in a separate location on a specific host (i.e. /var/log/dirsrv/slapd/access). I want to send these logs through logstash so I can filter the logs because I only need a few logs that are generated inside it instead of taking in all of the data

I installed logstash (on the same server as elasticsearch and kibana) and created a filter in /etc/logstash.conf.d/access.conf. But I wanted to send only these logs to logstash to get filtered while keeping the other logs (/var/log/secure.... etc....etc) forwarded straight to Elasticsearch. Is that possible?
Do I just add the output inside the filebeat.yml file located inside the elastic-agent path (/opt/Elastic/Agent/data/elastic-agent/install/filebeat/filebeat.yml) and enable the logstash.yml module or will that just send all logs (on that host machine) through logstash?

2

You can add in filebeat.yml per log params.
field.type = log1
field.type = log2
...

Then in LS, add lines:

if [field][type] =="log1"
.... # code to filter msges
if [field][type] =="log2 " {
drop {} # delete all logs from logs
}
3

A filebeat instance can only have one output, so you would need two filebeat instances. One to send stuff direct to elasticsearch and one to send stuff to logstash. See this thread.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.