Error when running pipelines.yml

Elastic Stack Logstash
1

Hello,

I'm a newbie in Elastic community and I'm trying to run my pipelines.yml on /etc/logstash/ using this command /usr/share/logstash/bin/logstash -f pipelines.yml but all I'm getting is the error below...

[WARN ] 2022-05-02 14:58:01.267 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[ERROR] 2022-05-02 14:58:06.399 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:384:in `block in converge_state'"]}

pipelines.yml

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: apache
  path.config: "/etc/logstash/conf.d/apache.conf"
  pipeline.workers: 1

apache.conf

input {
        kafka{
            bootstrap_servers => "ipaddress:port"
            topics => ["apache-topic"]
        }
}
filter {
         grok {
# GROK PATTERNS
        }
}
output {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "apache"
                user => "elastic"
                password => "password"
                }
        stdout{
                }
 }

I want to run the pipelines.yml since I think it will help me run the configuration files in just one instance as I have more than 10 configuration files. Thank you in advanced :slight_smile:

2

What command line are you using to start logstash?

3

Hello @Badger it is

systemctl start logstash

4

In that case we need to know what command line systemd is using to start the logstash service. In either the /lib/systemd/system or /etc/systemd/system directory there will probably be a file called something like logstash.service, and in that file I would expect there to be an ExecStart line that tells systemd how to start the program.

5

I think its running on /etc/systemd/system because when I checked the /lib/systemd/system there's no logstash.service available. As for the content of logstash.service in /etc/systemd/system hhehre it is...

logstash.service

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target
6

I am confused are you starting from the command line or systemd

That command above is manually starting logstash from the command line AND if you wanted to test you config then the command would be

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache.conf --path.settings /etc/logstash

Not pointing at the -f pipelines.yml

7

Hello @stephenb sorry for the confusion. What i'm trying to do here is that I'm trying to run the pipelines.yml since I have other configuration files and I want to run them in just one instance.

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:384:in `block in converge_state'"]}

Pasting the error whenever I''m trying to run the pipelines.yml.

In addition, thank you for clarifying this, now I know.

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache.conf --path.settings /etc/logstash

Thank you! :innocent:

8

So if you just run

/usr/share/logstash/bin/logstash --path.settings /etc/logstash

That should use the pipelines.yml file assuming it is in the proper location

The -f option is used to point to a specific pipeline config file when you don't use it logstash used the pipelines.yml

9

It worked! Thank you @stephenb :clap: :clap: :clap:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.