(seems) Doesn't read default configuration, logstash.conf

Elastic Stack Logstash
1

Hi,

Newbie here, just installed Logstash, have 2 questions.

  1. I follow this step, https://www.elastic.co/guide/en/logstash/5.0/config-examples.html, run in command line and working.

[/opt/logstash]$ bin/logstash -f /opt/logstash/logstash-access_log.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "x.x.x.x - - [17/Oct/2016:16:45:01 +0800] "GET /KG HTTP/1.1" 302 -",
"@version" => "1",
"@timestamp" => "2016-10-17T09:02:42.609Z",
"path" => "/myapp/logs/access_log.2016-10-17.txt",
"host" => "ip-y.y.y.y",
"type" => "apache_access"

I copied the logstash-access_log.conf into /etc/logstash/conf.d/logstash.conf (just rename into another file).

$ diff /opt/logstash/logstash-access_log.conf /etc/logstash/conf.d/logstash.conf | wc -l
0

But when I start logstash as a service, service logstash st,art, it doesn't send any data at all and the logs only show below,

[/var/log/logstash]$ cat *
{:timestamp=>"2016-10-17T11:53:29.162000+0800", :message=>"Pipeline main started"}
{:timestamp=>"2016-10-17T14:57:02.191000+0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2016-10-17T14:57:02.193000+0800", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2016-10-17T14:57:02.705000+0800", :message=>"Pipeline main has been shutdown"}
{:timestamp=>"2016-10-17T16:49:18.867000+0800", :message=>"Pipeline main started"}
{:timestamp=>"2016-10-17T17:00:01.734000+0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2016-10-17T17:00:01.736000+0800", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2016-10-17T17:00:02.033000+0800", :message=>"Pipeline main has been shutdown"}
Sending logstash logs to /var/log/logstash/logstash.log.
{:timestamp=>"2016-10-17T16:49:18.867000+0800", :message=>"Pipeline main started"}
{:timestamp=>"2016-10-17T17:00:01.736000+0800", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2016-10-17T17:00:02.033000+0800", :message=>"Pipeline main has been shutdown"}

[/var/log/logstash]$ ls -ltr
total 8
-rw-r--r-- 1 root root 0 Oct 17 16:49 logstash.err
-rw-r--r-- 1 root root 325 Oct 17 17:00 logstash.stdout
-rw-r--r-- 1 logstash logstash 774 Oct 17 17:00 logstash.log

What's wrong?

  1. The input log contains date which is rolling daily. How to make it auto update in configuration file? I tried to make like "/mylog/logs/access_log.date +"%Y-%m-%d".txt" is not working.

input {
file {
path => "/mylog/logs/access_log.2016-10-17.txt"
start_position => "beginning"
}
}

2
  1. There are many possible reasons why this doesn't work, e.g. that the user that Logstash runs as when run as a service (most likely "logstash") doesn't have permissions to read the access log. If you increase the logging verbosity by adjusting LS_OPTS in /etc/sysconfig/logstash or /etc/default/logstash (I believe that's how it works also in Logstash 5) you'll get additional details.
  2. Just use a filename pattern that selects all log files, like /mylog/logs/access_log.*.txt.
3

Fantastic, solves all my 2 problems !!! :clap:

Reason for no. 1, logstash is run under logstash user when I run as a service where logstash user doesn't have an READ access to my log file. I thought it's run under ec2-user or root.

Another question, how to make logstash auto reload configuration when runs as a service, instead of restarting? In command line I can use --auto-reload option.

4

Another question, how to make logstash auto reload configuration when runs as a service, instead of restarting? In command line I can use --auto-reload option.

Again, command line options are (AFAIK) controlled via LS_OPTS in /etc/sysconfig/logstash or /etc/default/logstash.

5

Thanks, it works!