Hello,
I'm currently working through upgrading a 2.6 ELK setup to 7.5 on a new server using what I can from the old config. The goal is to send rsyslog data from clients to Logstash. Previously, this worked (mostly) fine using the following Logstash Input, Filter, and Output config:
input {
tcp {
port => 1514
type => syslog
}
udp {
port => 1514
type => syslog
}
udp {
port => 1515
type => PFSense
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date
{
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
index => "logstash-%{+YYYY.MM.dd}"
document_type => "syslog"
}
}
In the updated VM, I'm using an rsyslog template to send the syslogs as JSON:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
And using an updated Logstash config to accommodate:
input {
tcp {
port => 1514
type => syslog
codec => json
}
tcp {
port => 514
type => syslog
codec => json
}
udp {
port => 1514
type => syslog
codec => json
}
udp {
port => 1515
type => PFSense
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
This mostly works well - we aren't getting grokparsefailures like before, the messages in Kibana are generally readable, and the fields are all relevant. You might notice there's no filter section - with the old filter in place on the new server, I was getting more grokparsefailure errors and it seems to be working fine (perhaps better) without any filter at all. However, every message has the same index, which is simply logstash. Previously, the index would have the date appended to the end. This is crucial, since we then use Curator to rotate old indices after a certain amount of time.
I've tried the following:
- Match date on "@timestamp" in the filter section
- Add the old filter back in its current state
- Set manage_template to false in the ES output section
- Remove the index line from the ES output section
Please let me know if there's any more info I can provide. I'm admittedly weak at Logstash and grok so this might be super obvious, but most of what I've been able to find online deals with either an old version of ELK or using Filebeats as inputs. Thank you.