Empty configuration for pipeline_id

Elastic Stack Logstash
1

Hi,

I am trying to configure for centralized pipeline management but failed. Would you please help check why? Thanks..

pipeline.yml

- pipeline.id: mainpipe
  path.config: "/etc/logstash/conf.d/*.conf"

ll path.config

-rw-r--r-- 1 root   root    165 Nov 26 10:44 /etc/logstash/conf.d/1_filebeat_input.conf
-rw-r--r-- 1 root   root   2397 Nov 26 10:44 /etc/logstash/conf.d/2_aa_json_filter.conf
-rw-r--r-- 1 root   root   4316 Nov 26 10:44 /etc/logstash/conf.d/2_channel_dst_filter.conf
-rw-r--r-- 1 root   root    193 Nov 26 10:44 /etc/logstash/conf.d/2_elk_demo.conf
-rw-r--r-- 1 root   root   3121 Nov 26 10:44 /etc/logstash/conf.d/2_olb_elk_filter.conf
-rw-r--r-- 1 root   root   1727 Nov 26 10:44 /etc/logstash/conf.d/2_safebanking_filter.conf
-rw-r--r-- 1 root   root   1696 Nov 26 10:44 /etc/logstash/conf.d/2_smartdss_filter.conf
-rw-r--r-- 1 root   root    197 Nov 26 10:44 /etc/logstash/conf.d/2_tde_filter.conf
-rw-r--r-- 1 root   root   1372 Nov 26 10:44 /etc/logstash/conf.d/2_z_channel_customerconnect_filter.conf
-rw-r--r-- 1 root   root   1476 Nov 26 10:44 /etc/logstash/conf.d/3_channel_customerconnect_filter.conf
-rw-rw-r-- 1 plam14 plam14 1735 Dec  7 15:54 /etc/logstash/conf.d/7_common_filter.conf
-rw-r--r-- 1 root   root    496 Dec 12 12:33 /etc/logstash/conf.d/9_elasticsearch_output.conf

logstash.yml

path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: logstash_system
xpack.monitoring.elasticsearch.url: "127.0.0.1:9200"
xpack.management.enabled: true
xpack.management.elasticsearch.username: logstash_internal
xpack.management.elasticsearch.password: logstash_internal
xpack.management.elasticsearch.url: "127.0.0.1:9200"
xpack.management.pipeline.id: ["mainpipe"]
xpack.management.logstash.poll_interval: 5s

logstash.service

[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target

Logstash Log

[2018-12-14T12:24:19,913][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["mainpipe"], :poll_interval=>"5000000000ns"}
[2018-12-14T12:24:22,476][INFO ][logstash.configmanagement.elasticsearchsource] Configuration Management License OK
[2018-12-14T12:24:23,124][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.3"}
[2018-12-14T12:24:24,957][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2018-12-14T12:24:24,959][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2018-12-14T12:24:25,064][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://logstash_internal:xxxxxx@127.0.0.1:9200/]}}
[2018-12-14T12:24:25,076][WARN ][logstash.configmanagement.elasticsearchsource] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@127.0.0.1:9200/"}
[2018-12-14T12:24:25,087][INFO ][logstash.configmanagement.elasticsearchsource] ES Output version determined {:es_version=>6}
[2018-12-14T12:24:25,088][WARN ][logstash.configmanagement.elasticsearchsource] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-12-14T12:24:25,172][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: mainpipe", :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:99:in `get_pipeline'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:85:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:84:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:150:in `converge_state_and_update'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:49:in `enable_monitoring'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:39:in `update_license_state'", "/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:39:in `setup_license_checker'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:23:in `initialize'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:98:in `after_agent'", "org/logstash/execution/EventDispatcherExt.java:69:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:355:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}

2

Turns out it's okay to have that error.
The noob mistake I had is that I didn't turn on xpack.security.enabled in kibana.yml.
When I turned it on, the error still persist in logstash log.
But when I created the pipeline mainpipe in Kibana UI (the pipeline I configured in logstash.yml), logstash will refresh itself and every works now!

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.