Configuring Multiple Pipelines

Elastic Stack Logstash
1

Logstash 6.1.1 on Server 2012 R2

Trying to configure multiple pipelines but coming up short. In pipelines.yml:

 - pipeline.id: Beats_Pipe
   path.config: "c:/Logstash/config/beats.yml"

Here's my beats.yml pipeline:

input {
  beats {
    port => 5044
    include_codec_tag => "false"
    id => "Beats_Input"
  }
}
output {
  elasticsearch {
    hosts => "ElasticSearch:9200"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

Debug when running bin/logstash (Omitted default LogStash Settings due to post character limitation):

[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *log.level: "trace" (default: "info")
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *http.host: "192.168.1.101" (default: "127.0.0.1")
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *http.port: 9600..9600 (default: 9600..9700)
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *queue.type: "persisted" (default: "memory")
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *queue.page_capacity: 134217728 (default: 262144000)
[2018-02-03T02:08:54,892][DEBUG][logstash.runner          ] *queue.max_bytes: 2147483648 (default: 1073741824)
[2018-02-03T02:08:54,907][DEBUG][logstash.runner          ] *path.queue: "C:/Buffer" (default: "C:/Logstash/data/queue")
[2018-02-03T02:08:54,907][DEBUG][logstash.runner          ] path.dead_letter_queue: "C:/Logstash/data/dead_letter_queue"
[2018-02-03T02:08:54,907][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2018-02-03T02:08:54,938][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"C:/Logstash/config/pipelines.yml"}
[2018-02-03T02:08:55,079][DEBUG][logstash.agent           ] Agent: Configuring metric collection
[2018-02-03T02:08:55,142][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2018-02-03T02:08:55,329][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2018-02-03T02:08:55,532][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2018-02-03T02:08:55,563][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] PeriodicPoller: Starting {:polling_interval=>5, :polling_timeout=>120}
[2018-02-03T02:08:55,673][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.1"}
[2018-02-03T02:08:55,704][DEBUG][logstash.agent           ] starting agent
[2018-02-03T02:08:55,767][DEBUG][logstash.agent           ] Starting puma
[2018-02-03T02:08:55,798][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2018-02-03T02:08:55,829][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"C:/Logstash/config/pipelines.yml"}
[2018-02-03T02:08:56,267][DEBUG][logstash.api.service     ] [api-service] start
[ERROR] 2018-02-03 02:08:56.345 [Ruby-0-Thread-1: C:\Logstash\lib\bootstrap\environment.rb:6] sourceloader - No configuration found in the configured sources.
[2018-02-03T02:08:56,376][DEBUG][logstash.agent           ] Converging pipelines
[2018-02-03T02:08:56,407][TRACE][logstash.agent           ] Converge results {:success=>true, :failed_actions=>[], :successful_actions=>[]}
[2018-02-03T02:08:56,438][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2018-02-03T02:08:56,454][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2018-02-03T02:08:56,454][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2018-02-03T02:08:56,470][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] PeriodicPoller: Stopping
[2018-02-03T02:08:56,548][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-02-03T02:09:01,737][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2018-02-03T02:09:01,737][DEBUG][logstash.agent           ] Converging pipelines
[2018-02-03T02:09:01,753][TRACE][logstash.agent           ] Converge results {:success=>true, :failed_actions=>[], :successful_actions=>[]}

If I run bin/logstash -f and specify the beats.yml pipeline, everything works properly. What am I doing wrong here?

2

Experiencing exactly the same issue, with the same version of Logstash as you. Did you manage to resolve this @wwalker?

3

Ya...accidently created two topics on this and answered it in the other thread. In the path.config, remove the drive letter. For example, instead of D:/logstash/config, make it /logstash/config. Yes, that's for Windows.

4

Great. Thanks very much @wwalker, that worked a treat.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.