Pipelines.yml still ignored

Elastic Stack Logstash
1

Greetings,
I have populated the pipelines.yml file but it is not getting picked up.
The goal being to separate the 2 pipelines i have running.
When i put them both in the /etc/logstash/conf.d they collide and logstash blends the fields from each pipeline, so i want to separate them.
But when i start up logstash as a service, as the log file shows, it just loops endlessly trying to load the apache and netflow modules. Any tips or pointers appreciated.
thx,james

Background Info

Section1 = Versions
Section2 = Contents of /etc/logstash/pipelines.yml
Section3 = Contents of /etc/logstash/logstash.yml
Section4 = Contents of /etc/systemd/system/logstash.service
Section5 = Contents /var/log/logstash/logstash-plain.log ( loops endlessly)

section 1
logstash.noarch 1:6.1.2-1 @elasticsearch-6.x
elasticsearch.noarch 6.1.2-1 @elasticsearch-6.x
kibana.x86_64 6.1.2-1 @elasticsearch-6.x

section 2
pipelines.yml

/etc/logstash# cat pipelines.yml
pipeline.id: flowlogs
path.config: "/etc/logstash/pipelines.d/flowlogs/flowlogs.conf"
pipeline.id: guestoslogs
path.config: "/etc/logstash/pipelines.d/guestos/guestoslogs.conf"

section 3
logstash.yml
Path config commented out
62 # Where to fetch the pipeline configuration for the main pipeline
63 #
64 #path.config: /etc/logstash/conf.d/*.conf
65 #

section 4
/etc/systemd/system/logstash.service
[root@zsmsazure logstash]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash

EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
#ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
ExecStart=/usr/share/logstash/bin/logstash
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[root@zsmsazure logstash]#

section 5
[2018-01-30T19:32:35,622][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-01-30T19:32:35,625][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-01-30T19:32:52,112][INFO ][logstash.modules.scaffold] Initializing module

2

It looks like the format of your pipelines.yml is wrong. Indentation is important with yaml. Here is an example that works...

- pipeline.id: process-ipfix
  queue.type: memory
  path.config: "/etc/logstash/pipelines/process/ipfix"

Notice the - before pipeline.id and two spaces before the lines beneath.

I have had a simple indentation error cause me problems as well. Hopefully this helps.

3

Thanks! will try this today.

4

Thanks Robert,
Much Appreciated.
That did the trick! My formatting was bad.
Now all 3 of my pipelines are running and there are no collisions.

cat pipelines.yml

  • pipeline.id: eventhubs
    queue.type: memory
    path.config: "/etc/logstash/pipelines.d/eventhubs/geteventhub.conf"
  • pipeline.id: flowlogs
    queue.type: memory
    path.config: "/etc/logstash/pipelines.d/flowlogs/flowlogs.conf"
  • pipeline.id: guestoslogs
    queue.type: memory
    path.config: "/etc/logstash/pipelines.d/oslogs/guestoslogs.conf"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.