Where to put pipeline configuration files on Windows 10?

Elastic Stack Logstash
1

Where should I put pipeline configuration files (input->filter->output) referred to in pipelines.yml file on Windows 10? I am using Logstash 6.2.2 and I noticed that if the file names do not contain accented characters (such as ş,ü etc.) the corresponding pipelines gets ignored completely when placed in the Logstash directory or its subdirectories.

In the documentation it says that these files must be put in /etc/logstash/conf.d directory in Linux based systems or else they will be ignored. However, there is no corresponding information on where to put these configuration files on Windows. I tried creating various directories under Logstash and other system location, yet these did not solve my problem. I would really appreciate it if someone can assist on this matter. Below is a snippet from the log file Logstash generated.

[2018-03-24T22:47:54,492][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[... "c:/Users/ongun.arisev/Downloads/ElasticStack/logstash-6.2.2/config/sql-pipeline.conf", "c:/Users/ongun.arisev/Downloads/ElasticStack/logstash-6.2.2/config/sql-pipeline_train.conf", ...]}

2

Where should I put pipeline configuration files (input->filter->output) referred to in pipelines.yml file on Windows 10?

I don't understand your question. pipelines.yml includes the path to the configuration files for each pipeline, so just place the files there. Logstash has no particular opinions about where the files are placed as long as the configuration is consistent with reality.

3

Yeah, you could argue that it says that, but that is not true. I don't think it is meant to say that.

The log message say those file name do not match the glob pattern. What glob pattern did you supply?

4

My files got ignored a couple of times that is why I asked this. Do you specify the glob pattern in the command line?
By the way the issue was remedied by omitting the drive letter or providing a relative path. Is this the glob pattern, could you provide an example?

5

No, you specify the glob pattern in the pipelines.yml. For example in

- pipeline.id: main
  path.config: "/etc/logstash/some.dir/*.conf"

the glob pattern is "/etc/logstash/some.dir/*.conf". And if there are any files in /etc/logstash/some.dir that do not match that pattern, e.g. /etc/logstash/some.dir/Foo.conf.bak, then it will include them in that log message.

If it is ignoring files that you think it should be using then your glob pattern is probably wrong.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.