Problems With configure Multi Pipelines Logtash V6.1.2

Elastic Stack Logstash
1

Hi, I trying configure logstash with multipipelines and cannot makes works, i follow the documentations and is doesnt work... I comment the path.config in the logstash.yml and add the path's of the each pipeline in pipelines.yml..

logstash.yml
#path.config: /usr/share/logstash/pipeline
path.data: /usr/share/logstash/
xpack.monitoring.elasticsearch.url: http://x.x.x.x:9200
xpack.monitoring.elasticsearch.username: xxxxxxxxxxx
xpack.monitoring.elasticsearch.password: xxxxxxxxxxx

pipelines.yml

  • pipeline.id: microservicios
    path.config: "/usr/share/logstash/pipeline/microservicios.conf"
    pipeline.workers: 1
  • pipeline.id: beats
    path.config: "/usr/share/logstash/pipeline/beats.conf"
    pipeline.workers: 1
  • pipeline.id: test
    pipeline.config: "/usr/share/logstash/pipeline/logstash.conf"

when i try up logstash service crash

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2018-02-22T23:55:22,468][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-02-22T23:55:22,498][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-02-22T23:55:23,989][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.1.2-java/modules/arcsight/configuration"}
[2018-02-22T23:55:27,435][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.2"}
[ERROR] 2018-02-22 23:55:27.681 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>ArgumentError, :message=>"Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in `get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:65:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `block in merge'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `merge'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:19:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:18:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2018-02-22T23:55:27,706][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Help Me please..

2

please remove quotes("") and try.

path.config: "/usr/share/logstash/pipeline/microservicios.conf"

path.config:/usr/share/logstash/pipeline/microservicios.conf ------try this

3

Thank you, I Remove (") and not work.

Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2018-02-23T15:21:08,351][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-02-23T15:21:08,385][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-02-23T15:21:10,143][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.1.2-java/modules/arcsight/configuration"}
[2018-02-23T15:21:13,766][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.2"}
[ERROR] 2018-02-23 15:21:14.013 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>ArgumentError, :message=>"Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in `get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:65:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `block in merge'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `merge'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:19:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:18:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2018-02-23T15:21:14,043][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

4

For the test pipeline you have pipeline.config where you should have path.config.

5

paste ur config file here???

6

Logstash.yml

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
#path.data: /usr/share/logstash/
xpack.monitoring.elasticsearch.url: http://x.x.x.x:9200
xpack.monitoring.elasticsearch.username: xxxxxxx
xpack.monitoring.elasticsearch.password: xxxxxxxx

Logstash.conf

input{
tcp {
type => "test"
port => 9900
codec => "json"
}
}
output {
if [type] == "test"{
elasticsearch {
hosts => ["x.x.x.x:9200"]
user => xxxxxxx
password => xxxxxx
index => "prueba"
}
}
stdout {
codec => rubydebug
}
}

Beats.conf

input {
beats {
port => 9400
type => "beats"
}
}
output {
if [type] == "beats"{
elasticsearch {
hosts => ["x.x.x.x:9200"]
user => xxxxx
password => xxxxxx
index => "beats-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["x.x.x.x:9200"]
user => xxxxx
password => xxxxx
index => "was-%{+YYYY.MM.dd}"
}
}
stdout {
codec => rubydebug
}
}

microservicios.yml

input{
tcp {
type => "microservices"
port => 9500
codec => multiline{
pattern => "^(?%{MONTHDAY}[-]%{MONTHNUM}[-]%{YEAR})|"
negate => true
what => "previous"
}
}
tcp {
type => "was"
port => 9700
codec => multiline{
pattern => "^(?%{MONTHDAY}[-]%{MONTHNUM}[-]%{YEAR})"
negate => true
what => "previous"
}
}
tcp {
type => "monitoring"
port => 18080
}
tcp {
type => "atla_services"
port => 9800
codec => "json"
}
}
filter {
if [type] == "microservices"{
grok {
match => { "message" => "(?%{MONTHDAY}[-]%{MONTHNUM}[-]%{YEAR})?|(?%{HOUR}:%{MINUTE}:%{SECOND})?|(%{NONNEGINT:sss})?|(%{GREEDYDATA:cic})?|(%{GREEDYDATA:idc})?|(%{UUID:sesUID})?|(%{UUID:trnUID})?|(%{NONNEGINT:opnNro})?|(%{NONNEGINT:opnNroHost})?|(%{IP:servIp})?|(%{HOSTNAME:servNom})?|(%{GREEDYDATA:class})?|(%{NONNEGINT:idClass})?|(%{LOGLEVEL:loglevel})?|(%{GREEDYDATA:namespace}) ?|(%{WORD:method})?|(%{GREEDYDATA:msgMicroservices})?"}
}
}
if [type] == "was"{
grok {
match => { "message" => "%{DATE:fecha_was}? %{TIME:hora_was}? [(%{GREEDYDATA:cic})?] [(%{GREEDYDATA:session})?] : [%{LOGLEVEL:loglevel}] (%{GREEDYDATA:namespace}) ? - %{GREEDYDATA:msgWas}?" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
if [type] == "atla_services"{
elasticsearch {
hosts => ["x.x.x.x.x:9200"]
user => xxxxxx
password => xxxxxx
index => "atlas-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["x.x.x.x:9200"]
user => xxxxxxx
password => xxxxxxxx
}
stdout {
codec => rubydebug
}
}
}

pipelines.yml

     - pipeline.id: microservicios
       path.config: /usr/share/logstash/pipeline/microservicios.conf
       pipeline.workers: 1
     - pipeline.id: beats
       path.config: /usr/share/logstash/pipeline/beats.conf
       pipeline.workers: 1
     - pipeline.id: test
       pipeline.config: /usr/share/logstash/pipeline/logstash.conf

Thats all my config

7

I read some internet forums and somebody suggets comment the path.config in losgtash.yml to force logstash read the content in pipelines.yml.. i show the logs... with that line commented in logs not appear ingnore pipelines.yml

[2018-02-26T20:49:15,934][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.2"}
[ERROR] 2018-02-26 20:49:16.183 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>ArgumentError, :message=>"Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in `get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:65:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `block in merge'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:84:in `merge'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:19:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:18:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2018-02-26T20:49:16,211][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Setting "pipeline.config" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

thanks

8

http.host: "0.0.0.0"
<path.config: /usr/share/logstash/pipeline ----u shud give the path of your config file here like ur config file is Logstash.conf then provide the path of this file.>

9

Ok, thats it the mistake... I sorry i can't believe it... i don't see the config with attention

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.