Hello All
I'm fairly new at Logstash and I was tasked of taking our old beast of a Logstash server into the new age.
We use our Filebeat/Logstash setup for a central logging server and nothing more.
I've got 90% of the way there now with Filebeat 7.1.1 and logstash 7.1.1 (from Filebeat 1.3.1 and logstash 2.3.x) however I'm stumbling over the last hurdle and could use a little bit of help.
We had a set of filters on the logstash end that would filter the logs into folders
The folders sorted arranged the logs by type then by year-month/date/server/filename.
EG: /logs/squid/2019-06/20/proxy1/access.log
Where I'm struggling is the formatting has changed quite a bit, I got the year-month and day working but I can't seem to get the server name and filename working anyone, what I'm left with is an output like the below.
EG: /logs/squid/2019-06/20/%{host}/%filename
Which is no good at all.
Here is what we use to have:
OLD CODE!
ruby {
code => "event['filename'] = event['source'].split('/').last"
}
ruby {
code => "event['index_day'] = event.timestamp.time.localtime.strftime('%d-%m-%y')"
}
ruby {
code => "event['index_month'] = event.timestamp.time.localtime.strftime('%Y-%m')"
}
ruby {
code => "event['index_day_only'] = event.timestamp.time.localtime.strftime('%d')"
}
output {
if [type] == "syslog" {
file {
path => "/logs/syslog/%{index_month}/%{index_day_only}/%{host}/%{filename}"
codec => line { format => "%{[message]}"}
}
}
}
And here is the new code:
NEW CODE
ruby {
code=> "event.set('filename', event.get('source').split('/').last)"
#Below is the old code tried both neither worked
#code => "event['filename'] = event['source'].split('/').last"
}
output {
if [fields][document_type] == "syslogtest" {
file {
path => "/logs/syslog/%{+YYYY-MM}/%{+dd}/%{host}/%{filename}"
codec => line { format => "%{[message]}"}
}
}
}
I tried to do some googling but I just couldn't get there.
I'm know I'm close I'm just not really understand what I'm missing, any help would be amazing thank you in advance.