Followed this tutorial for a basic install and configure the ELK stack for ingesting syslogs.
My goal is to create a system where I can ingest apache logs from wordpress sites that use wpengine. WPengine won't allow you to install filebeats so you have to regularly request the log files and download them. This means I want to configure this so that I can download a file to a directory and ELK will automatically ingest the log file.
Per the tutorials instructions, these are my config files that are currently working and ingesting syslogs ( again, which I want to change to ingest apache logs that I download from wpengine)
Inside:
/etc/logstash/conf.d/
I've added three config files titled:
02-beats-input.conf
10-syslog-filter.conf
30-elasticsearch-output.conf
The contents of these files are respectively:
02-beats-input.conf
input {
beats {
port => 5044
}
}
10-syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Then finally, I changed the filebeats .yml like this:
/etc/filebeat/filebeat.yml
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/auth.log
- /var/log/syslog
#- /var/log/*
#- c:\programdata\elasticsearch\logs\*
I found this apache configuration example that i'm using. I tried to change the configuration files to accept a single apache log from stdin just to test that I'm effectively changing the config/ingestion pipeline but it's not working.
One big question I have, is how does ELK know how to select from the three different config files? For example, the name "02-beats-input.conf". Is it the "02", or the "input" in the file name that is telling elk which files to use for configuration? Or is it just grabbing whatever files it finds in that directory and applying those instructions?
If that's the case it may explain why it's breaking since I'm saving the 'working' config files from the tutorial under a new name in that same directory and adding new ones with the changes.
Thanks!