Fairly new to ELK Stack so please bear with me
I have a single beats.conf file in Logstash at the moment for a new deployment and I thought it was all working fine, however when trying to make some changes earlier I noticed my filtering hasn't been working correctly. I have syslog, secure and trying to push apache logs through, so I have my filter setup with if and else if based on a field type I have defined in my filebeat config. EG the below.
My filebeat.yml
- input_type: log
paths:
- /var/log/messages*
- /var/log/syslog*
ignore_older: 1h
fields:
logtype: syslog_data
Then in logstash beats.conf
filter {
if [fields][log_type] =~ "syslog_data" {
That all fails but parses out correctly because of my final else statement.
BUT, if I change the filebeat.yml to
- input_type: log
paths:
- /var/log/messages*
- /var/log/syslog*
ignore_older: 1h
document_type: syslog
fields:
logtype: syslog_data
and my beats.conf filter to
filter {
if [type] == "syslog" {
Then it all works? What I am I doing wrong? I thought fields was the best type to use as I get deprecated warnings for document_type? If I can filter on my field type then I think everything should work for me?