Hello,
I'm new with ELK and i start to configure a cluster
3 elasticsearch nodes
1 Kibana
2 logstash nodes
without SSL the configuation is good without problem, put after setting SSL, TLS et HTTPS refering to https://www.elastic.co/fr/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash I have some problem with logstash.
Mar 24 10:23:41 ls-node-01 systemd[1]: Started logstash.
Mar 24 10:24:02 ls-node-01 logstash[14701]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Mar 24 10:24:03 ls-node-01 logstash[14701]: [2020-03-24T10:24:03,635][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Mar 24 10:24:03 ls-node-01 logstash[14701]: [2020-03-24T10:24:03,793][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.0"}
Mar 24 10:24:06 ls-node-01 logstash[14701]: [2020-03-24T10:24:06,246][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
Mar 24 10:24:06 ls-node-01 logstash[14701]: [2020-03-24T10:24:06,251][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
Mar 24 10:24:06 ls-node-01 logstash[14701]: [2020-03-24T10:24:06,977][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 15, column 14 (byte 451) after output {\n elasticsearch {\n hosts => [\"https://10.14.0.150:9200\", \"https://10.14.0.151:9200\"]\n cacert => '/etc/logstash/config/certs/ca.crt'\n user => 'logstash_writer'\n password ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
Mar 24 10:24:07 ls-node-01 logstash[14701]: [2020-03-24T10:24:07,288][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 15, column 14 (byte 451) after output {\n elasticsearch {\n hosts => [\"https://10.14.0.150:9200\", \"https://10.14.0.151:9200\"]\n cacert => '/etc/logstash/config/certs/ca.crt'\n user => 'logstash_writer'\n password ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,492][INFO ][org.reflections.Reflections] Reflections took 54 ms to scan 1 urls, producing 20 keys and 40 values
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,681][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", ssl_certificate_verification=>false, password=><password>, hosts=>[https://10.14.0.150:9200], cacert=>"/etc/logstash/config/certs/ca.crt", sniffing=>false, manage_template=>false, id=>"3542a896731687cee6eb1e5889877a286c7c466e27d7fe58174ca1d46fb3229a", user=>"logstash_system", ssl=>true, document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_5b593356-e666-4271-b0a2-699ef6c1454f", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,802][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
Mar 24 10:24:08 ls-node-01 logstash[14701]: ** WARNING ** You have enabled encryption but DISABLED certificate verification.
Mar 24 10:24:08 ls-node-01 logstash[14701]: ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,889][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@10.14.0.150:9200/]}}
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,952][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@10.14.0.150:9200/"}
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,965][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>7}
Mar 24 10:24:08 ls-node-01 logstash[14701]: [2020-03-24T10:24:08,968][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
Mar 24 10:24:09 ls-node-01 logstash[14701]: [2020-03-24T10:24:09,039][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://10.14.0.150:9200"]}
Mar 24 10:24:09 ls-node-01 logstash[14701]: [2020-03-24T10:24:09,147][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x29d32722 run>"}
Mar 24 10:24:10 ls-node-01 logstash[14701]: [2020-03-24T10:24:10,109][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
Mar 24 10:24:10 ls-node-01 logstash[14701]: [2020-03-24T10:24:10,484][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Mar 24 10:24:11 ls-node-01 logstash[14701]: [2020-03-24T10:24:11,296][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
Mar 24 10:24:12 ls-node-01 logstash[14701]: [2020-03-24T10:24:12,129][INFO ][logstash.runner ] Logstash shut down.
Mar 24 10:24:12 ls-node-01 systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Mar 24 10:24:12 ls-node-01 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 2.
Mar 24 10:24:12 ls-node-01 systemd[1]: Stopped logstash.
Mar 24 10:24:12 ls-node-01 systemd[1]: Started logstash.
logstash.yml
node.name: ls-node-01
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'changeme'
xpack.monitoring.elasticsearch.hosts: [ 'https://10.14.0.150:9200' ]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt
for test i use this config file
cat /etc/logstash/conf.d/demo.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => '/etc/logstash/config/certs/ca.crt'
ssl_certificate => '/etc/logstash/config/certs/ls-node-01.crt'
ssl_key => '/etc/logstash/config/certs/ls-node-01.pkcs8.key'
}
}
output {
elasticsearch {
hosts => ["https://10.14.0.150:9200", "https://10.14.0.151:9200"]
cacert => '/etc/logstash/config/certs/ca.crt'
user => 'logstash_writer'
password 'changme'
#manage_template => false
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#index => 'logstash-%{+YYYY.MM.dd}'
}
}
Bellow version for logstash and elasticsearch
/usr/share/elasticsearch/bin/elasticsearch -version
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 7.6.0, Build: default/deb/7f634e9f44834fbc12724506cc1da681b0c3b1e3/2020-02-06T00:09:00.449973Z, JVM: 13.0.2
/usr/share/logstash/bin/logstash --version
logstash 7.6.0
all node are deployed on ubuntu 18.04.
thank you