Hi,
I am trying to configure for centralized pipeline management but failed. Would you please help check why? Thanks..
pipeline.yml
- pipeline.id: mainpipe path.config: "/etc/logstash/conf.d/*.conf"
ll path.config
-rw-r--r-- 1 root root 165 Nov 26 10:44 /etc/logstash/conf.d/1_filebeat_input.conf -rw-r--r-- 1 root root 2397 Nov 26 10:44 /etc/logstash/conf.d/2_aa_json_filter.conf -rw-r--r-- 1 root root 4316 Nov 26 10:44 /etc/logstash/conf.d/2_channel_dst_filter.conf -rw-r--r-- 1 root root 193 Nov 26 10:44 /etc/logstash/conf.d/2_elk_demo.conf -rw-r--r-- 1 root root 3121 Nov 26 10:44 /etc/logstash/conf.d/2_olb_elk_filter.conf -rw-r--r-- 1 root root 1727 Nov 26 10:44 /etc/logstash/conf.d/2_safebanking_filter.conf -rw-r--r-- 1 root root 1696 Nov 26 10:44 /etc/logstash/conf.d/2_smartdss_filter.conf -rw-r--r-- 1 root root 197 Nov 26 10:44 /etc/logstash/conf.d/2_tde_filter.conf -rw-r--r-- 1 root root 1372 Nov 26 10:44 /etc/logstash/conf.d/2_z_channel_customerconnect_filter.conf -rw-r--r-- 1 root root 1476 Nov 26 10:44 /etc/logstash/conf.d/3_channel_customerconnect_filter.conf -rw-rw-r-- 1 plam14 plam14 1735 Dec 7 15:54 /etc/logstash/conf.d/7_common_filter.conf -rw-r--r-- 1 root root 496 Dec 12 12:33 /etc/logstash/conf.d/9_elasticsearch_output.conf
logstash.yml
path.data: /var/lib/logstash path.logs: /var/log/logstash xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: logstash_system xpack.monitoring.elasticsearch.url: "127.0.0.1:9200" xpack.management.enabled: true xpack.management.elasticsearch.username: logstash_internal xpack.management.elasticsearch.password: logstash_internal xpack.management.elasticsearch.url: "127.0.0.1:9200" xpack.management.pipeline.id: ["mainpipe"] xpack.management.logstash.poll_interval: 5s
logstash.service
[Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target
Logstash Log
[2018-12-14T12:24:19,913][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["mainpipe"], :poll_interval=>"5000000000ns"}
[2018-12-14T12:24:22,476][INFO ][logstash.configmanagement.elasticsearchsource] Configuration Management License OK
[2018-12-14T12:24:23,124][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.3"}
[2018-12-14T12:24:24,957][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2018-12-14T12:24:24,959][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2018-12-14T12:24:25,064][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://logstash_internal:xxxxxx@127.0.0.1:9200/]}}
[2018-12-14T12:24:25,076][WARN ][logstash.configmanagement.elasticsearchsource] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@127.0.0.1:9200/"}
[2018-12-14T12:24:25,087][INFO ][logstash.configmanagement.elasticsearchsource] ES Output version determined {:es_version=>6}
[2018-12-14T12:24:25,088][WARN ][logstash.configmanagement.elasticsearchsource] Detected a 6.x and above cluster: thetypeevent field won't be used to determine the document _type {:es_version=>6}
[2018-12-14T12:24:25,172][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: mainpipe", :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:99:in `get_pipeline'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:85:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:84:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:150:in `converge_state_and_update'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:49:in `enable_monitoring'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:39:in `update_license_state'", "/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:39:in `setup_license_checker'", "/usr/share/logstash/x-pack/lib/monitoring/internal_pipeline_source.rb:23:in `initialize'", "/usr/share/logstash/x-pack/lib/monitoring/monitoring.rb:98:in `after_agent'", "org/logstash/execution/EventDispatcherExt.java:69:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:355:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}