I installed on CentOS-7 an ELK stack according to a tutorial. This worked all fine and I can see and search the logs of the localhost server itself in the Kibana interface.
The purpose of the server is to act as a centralized remote log sever with for rsyslog. So, I used one of my regular machines to install and configure rsyslog there to use UDP port 514:
/etc/rsyslog.conf
<>
Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
$ModLoad imudp
$UDPServerRun 514
</>
With tcpdump on the ELK-server I can see the packets are incoming:
tcpdump -i p3p2 | grep ekgen7
This guide mentions to create template files on the ELK logstash server (ekgen9):
vim /etc/rsyslog.d/70-output.conf
# This line sends all lines to defined IP address at port 10514
# using the json-template format.
*.* @127.0.0.1:10514;json-template
vim /etc/rsyslog.d/01-json-template.conf
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
The problem is:
But there are still no messages shown in Kibana.
I might have here a problem with the logstash filter. Is that the correct port (10514)? Where does this "port shift" occur? I am not sure how the ports belonging to logstash are chosen.
curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'
doesn't show anything about rsyslog, I only see:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}
Another web page suggested to create another logstash.conf, defining an input filter for port 10514:
vim /etc/logstash/conf.d/logstash.conf
input {
udp {
host => "127.0.0.1"
port => 10514
codec => "json"
type => "rsyslog"
}
}
# The Filter pipeline stays empty here, no formatting is done. filter { }
# Every single log will be forwarded to ElasticSearch. If you are using another port, you should specify it here.
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}
}
Is 9200 the default port for Elastichsearch? At least I defined it to this in
an Elasticsearch output filter, following the first guide:
vim /etc/logstash/conf.d/30-Elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
But that only seems to pick up the localhost logs.
I am pretty sure that I am doing a simple/stupid mistake here. I am entirely new to ELK - not to system administration, but up to now delved in local logs.
For security reasons I want to establish this centralized log server, and I am all in favour of open source, so I chose ELK.
Any help is highly appreciated.
Best wishes,
Sven