Rsyslog collection using Logstash

Elastic Stack Logstash
1

Hi All,

I was hoping for some help setting up syslog collection using the syslog input plugin for Logstash on Ubuntu 14.04 LTS.

We have configured Topbeat, Logstash, Elasticsearch and Kibana and that seems to be working nicely (except for disk and file system metrics, but will start a separate thread regarding that).

I had a look at the logstash reference and did not find that very intuitive for installation and configuration.

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html

I am hoping that our use case is relatively simple and many other users will have implemented syslog input plugin on Ubuntu.

I understand syslog is the comprehensive of the system logs, we really want to be collecting as much as we sensibly can to monitor the server and security.

I am not sure if we need to be collecting the auth.log separately using Filebeat or Logstash, would welcome any recommendations there as well.

Hope to hear from you soon.

2

I had a look at the logstash reference and did not find that very intuitive for installation and configuration.

So... what's unclear? What have you tried? What happened? What did you expect should've happened?

I am not sure if we need to be collecting the auth.log separately using Filebeat or Logstash, would welcome any recommendations there as well.

You can set up your syslog daemon to send all messages via the syslog protocol to a remote server, including what usually goes to auth.log.

3

Alright, ready to delve in again..

Here is the example configuration file,

https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_syslog_messages

`input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}`

Stop the logstash service 'sudo service logstash stop'

Update the configuration file and then test the configuration 'bin/logstash -f logstash-syslog.conf'

Last time got into trouble with testing configuration files, what's the right way to approach this?

/opt/logstash/bin/logstash -f logstash-syslog.conf

This will use the logstash-syslog.conf file in /opt/logstash/bin/ ?

After successful foreground testing, update the logstash.conf file in etc/logstash/conf.d/

Then start the service in the background?

4

Last time got into trouble with testing configuration files, what's the right way to approach this?

We can't explain your troubles unless you explain what they were.

This will use the logstash-syslog.conf file in /opt/logstash/bin/ ?

Yes, if your current directory if /opt/logstash/bin.

5

The issue seemed to be if there was more than one configuration file in the directory for the back ground service. Multiple files (with different names) seemed to be why the service was stopping

6

If you point Logstash to a directory it'll read all files in the directory. This can cause interesting behavior if you're not prepared for it.

7

Okay, good progress with this plugin and went through the configuration and test process, solved issues as they came up.

https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_syslog_messages

Telnet tested in foreground

bin/logstash -f logstash-syslog.conf

Updated .conf

/etc/logstash/conf.d/logstash.conf

Started service and seems to be running.

However, checking the indices cannot see an index for syslog (or collectd)?

curl 'localhost:9200/_cat/indices?v'

Topbeat and Kibana installations created these automatically when the service started, what's the situation for syslog (and collectd)?

8

Let's not discuss the same thing in multiple threads. Please stick to Disk and file system usage collection using Logstash.

9

Good point, understood. Will keep each thread focused and not duplicate discussion.