Logstash is not loading the filter conf files

kittu9999 · July 17, 2020, 5:40pm

HI Team,

we have upgrade our ELK stack from V5.0.1 to V6.8.1. Basically we moved to docker containers. we are having an issue with new logstash and the necessary fields are not showing in kibana.

we have the same conf files from old logstash (version 5.0.1) to new logstash (6.8.1 version), after that fields are not creating. so i have enabled the trace and found the conf files are skipping. Can you please help me here?

[2020-07-17T16:29:43,836][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf/test.conf", "/etc/logstash/conf/test1.conf", "/etc/logstash/conf/test2.conf", "/etc/logstash/conf/test3.conf"]}

Badger · July 17, 2020, 5:44pm

Are you using pipelines.yml? If so, what does it look like? If not, what is path.config set to? (The debug messages should include it.)

kittu9999 · July 17, 2020, 5:47pm

i have set path.config as mentioned below

path.config: /etc/logstash/conf

kittu9999 · July 17, 2020, 7:11pm

Here is DEBUG logs

gist.github.com

https://gist.github.com/kn42447/2a0828babb56aa3e4ec27c9fa81c0eee

logstash_trace_log

bash-4.2# cat logstash-plain.log
[2020-07-17T18:53:08,489][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/etc/logstash/modules/netflow/configuration"}
[2020-07-17T18:53:08,494][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6f7a7030 @directory="/etc/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2020-07-17T18:53:08,496][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/etc/logstash/modules/fb_apache/configuration"}
[2020-07-17T18:53:08,496][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x6ad57ee3 @directory="/etc/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2020-07-17T18:53:08,541][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2020-07-17T18:53:08,550][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2020-07-17T18:53:08,856][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2020-07-17T18:53:08,856][DEBUG][logstash.runner          ] node.name: "ip-10-91-132-138.tr1stg.clstage.wellmanage.com"
[2020-07-17T18:53:08,857][DEBUG][logstash.runner          ] *path.config: "/etc/logstash/conf/logstash.conf"

This file has been truncated. show original

Badger · July 17, 2020, 7:35pm

Not so. The debug logs show

[logstash.runner          ] *path.config: "/etc/logstash/conf/logstash.conf"

So it looks through the /etc/logstash/conf directory and tells you it is ignoring everything except logstash.conf

kittu9999 · July 17, 2020, 7:37pm

Yeah, correct. is there any other i can fix this issue?

Badger · July 17, 2020, 7:40pm

Change the setting of path.config?

kittu9999 · August 7, 2020, 3:43am

i have included our filter configuration in /etc/logstash/conf/logstash.conf, and my issue is resolved. Thanks!

system · September 4, 2020, 3:44am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logs are not visible in Kibana Elasticsearch docker	4	203	March 23, 2024
Config filter not found when using Logstash docker image Logstash docker	3	635	April 7, 2020
Logstash conf not loading new version Logstash	4	770	September 26, 2018
Index not creating Logstash	18	844	September 23, 2021
Logstash Not Loading Config Files Running as Service Logstash	1	441	April 20, 2022

Logstash is not loading the filter conf files

Related topics