No Output Until Verbose

[johnslippers (John)]

Hi guys,

This is a very strange one. I am unable to get logstash to output to file by starting logstash as normal:

start logstash

The strange thing is, when I start logstash in verbose mode it works:

/usr/share/logstash/bin/logstash --path.settings=/etc/logstash --verbose

If I start it in verbose mode it gives me an error but still works:

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

The log4j2 file is located directly under "/etc/logstash/". Anyway, like I said it works. So starting logstash normally I just get a bunch of the following in the log file:
[DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[DEBUG][logstash.inputs.file ] globbed_files: /root/scserv/config/2411/w3clog1.log: glob is: []

I am using logstash 5.4 and the input file is a non-changing test file I created to simulate the environment I want to set this up on. Here is my conf file:

input {
_ file {_
_ path => "/root/scserv/config/2411/w3clog1_.log"_
_ start_position => beginning_
_ sincedb_path => "/dev/null"_
_ ignore_older => 0_
_ }_
}

filter {
_ grok {_
_ match => { _
_ "message" => '%{IPV4:ip_address} %{IPV4:src_ip} %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time} %{NOTSPACE:stream} %{NUMBER:c_reply} %{NOTSPACE:user_agent} %{NUMBER:sc_bytes} %{NUMBER:x_duration} %{NUMBER:avgbandwidt}'_
_ }_
_ }_
_ grok {_
_ match => { _
_ "path" => '%{GREEDYDATA}/%{GREEDYDATA}/%{GREEDYDATA}/%{GREEDYDATA:sid}/%{GREEDYDATA}.log'_
_ }_
_ }_
_ geoip {_
_ source => "src_ip"_
_ target => "geoip"_
_ database => "/usr/share/logstash/vendor/geoip/GeoLite2-City.mmdb"_
_ }_
_ mutate {_
_ add_field => {_
_ "date" => "%{year}-%{month}-%{day} %{time}"_
_ }_
_ add_field => {_
_ "country" => "%{[geoip][country_name]}"_
_ }_
_ remove_field => ["year", "month", "day", "time", "message", "@timestamp", "@version", "host", "c_reply", "geoip", "path", "src_ip", "stream"]_
_ } _
}

output {
_ file {_
_ path => "/root/test.log"_
_ }_
}

"/root/test.log" only gets filled with verbose mode on.

[magnusbaeck (Magnus Bäck)]

This is a very strange one. I am unable to get logstash to output to file by starting logstash as normal:

start logstash

The strange thing is, when I start logstash in verbose mode it works:

/usr/share/logstash/bin/logstash --path.settings=/etc/logstash --verbose

This has nothing to do with the level of logging. In the first case Logstash will run as the logstash user and in the second case it'll run as whatever user is running in your shell.

[DEBUG][logstash.inputs.file ] globbedfiles: /root/scserv/config/2411/w3clog1.log: glob is: _

The logstash user doesn't have access to /root so the filename pattern expands to an empty list and hence Logstash has nothing to do. Why are you storing log files there anyway?

There's been an identical thread here in the past 24 hours.

[johnslippers (John)]

Thanks Magnus. You have hit the nail on the head. After changing the location for both my input and output files to a place where the logstash user had access to, it started working.