Logstash starts successfully but doesn't generate events

Hello,

Logstash starts successfully but doesn't generate events when I change path to include subdirectories.

My logstash.conf looks like:-

input {
file {
path => "/local/kdmlogs/2019031905/*/*"
exclude => "*.s3logs"
start_position => "beginning"
sincedb_path => "/tmp/.sincedb"
tags => ["logs"]
}
}

filter {
if "logs" in [tags] {
grok {
match => {
"message" => "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:date}:%{HOUR}%{MINUTE}(%{SECOND}) %{SYSLOGPROG}:%{GREEDYDATA:syslog_message}"
}
}

grok {
  match => {
    "path" => "/local/kdmlogs/2019031905/*/%{GREEDYDATA}#%{GREEDYDATA:device_type}#%{GREEDYDATA}#%{GREEDYDATA:dsn}#%{GREEDYDATA:s3key}"
  }
}

#Remove fields we don 't want (to save space)
mutate {
  remove_field => ["message", "path", "host"]
}

}
metrics {
meter => "events"
add_tag => "metric"
}
}

output {
if "logs" in [tags] {
if ([year]) {
stdout{}
}
}
if "metric" in [tags] {

file {
  path => "/local/kdmlogs/metric/test-metric.txt"
  codec => line {
    format => "rate: %{[events][rate_1m]}"
  }
}

}
}

In the input, under file, when I change the path to :-
path => "/local/kdmlogs/2019022022/1/*"
i.e. all files under subdirector "1" under directory "2019022022", it works well.

But with the above logstash config, the logstash starts but doesn't generate any events. I remember doing the same thing some time back and it worked well previosuly.

When I add a --debug, all I see is:-
[DEBUG] 2019-03-20 12:32:41.365 [Ruby-0-Thread-4: :1] file - Starting flush cycle
[DEBUG] 2019-03-20 12:32:43.365 [Ruby-0-Thread-4: :1] file - Starting flush cycle
[DEBUG] 2019-03-20 12:32:44.234 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

Can someone please help me with same? I've tried almost all the solutions posted on the internet.

This is only observed in Logstash 6.4.0 and works well in logstash 6.2.4
Thanks,
Yogendra

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.