I am trying to secure between Filebeat and Logstash. However, I do encounter some difficulties.
Without SSL the connection does work.
First I created with the Elasticsearch-certuitil a CA.
C:\PATH\elasticsearch-8.0.1>bin\elasticsearch-certutil ca --pem
This gave me two files in a zip. ca.crt and ca.key
I created with those two files a certificate with a key.
C:\PATH\elasticsearch-8.0.1>bin\elasticsearch-certutil cert --ca-cert C:\PATH\elasticsearch-8.0.1\ca\ca.crt --ca-key C:\PATH\elasticsearch-8.0.1\ca\ca.key
This command gave me a zip which contains instance.key and instance.crt
I copied all files to logstash and the server, where Filebeat is running. And did set it up like this:
My Logstash conf:
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["C:\PATH\logstash-8.0.1\config\certs\ca.crt"]
ssl_certificate => "C:\PATH\logstash-8.0.1\config\certs\instance.crt"
ssl_key => "C:\PATH\logstash-8.0.1\config\certs\instance.key"
ssl_verify_mode => "force_peer"
}
}
Filebeat.yml:
output.logstash:
hosts: ["IPADRESS:5044"]
ssl.certificate_authorities: ["C:\\PATH\\ca.crt"]
ssl.certificate: "C:\\PATH\\instance.crt"
ssl.key: "C:\\PATH\\instance.key"
Filebeat is not giving any error messages.
Logstash is giving me following: File does not contain valid private key. How am I getting this to work? What am I doing wrong?
C:\PATH\logstash-8.0.1>bin\logstash.bat -f probe-pipeline6.conf --config.reload.automatic
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/PATH/logstash-8.0.1/logs which is now configured via log4j2.properties
[2022-05-27T15:26:26,884][INFO ][logstash.runner ] Log4j configuration path used is: C:\PATH\logstash-8.0.1\config\log4j2.properties
[2022-05-27T15:26:26,899][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-05-27T15:26:26,908][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"}
[2022-05-27T15:26:26,911][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-05-27T15:26:27,022][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-05-27T15:26:29,059][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-05-27T15:26:31,828][INFO ][org.reflections.Reflections] Reflections took 78 ms to scan 1 urls, producing 120 keys and 417 values
[2022-05-27T15:26:34,002][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-05-27T15:26:34,049][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:34,362][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:34,764][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:34,811][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:34,826][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:34,873][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:34,889][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:34,889][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:34,967][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:34,967][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:35,014][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:35,108][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:35,123][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:35,123][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:35,123][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:35,123][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:35,123][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:35,139][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:35,139][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:35,170][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:35,233][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:35,233][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:35,233][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:35,248][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:35,248][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:35,248][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:35,279][WARN ][logstash.filters.grok ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-05-27T15:26:35,294][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:35,681][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/PATH/logstash-8.0.1/probe-pipeline6.conf"], :thread=>"#<Thread:0x25e6671 run>"}
[2022-05-27T15:26:36,851][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.16}
[2022-05-27T15:26:36,882][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-05-27T15:26:37,023][ERROR][logstash.inputs.beats ][main] SSL configuration invalid {:exception=>Java::JavaLang::IllegalArgumentException, :message=>"File does not contain valid private key: C:\\PATH\\logstash-8.0.1\\config\\certs\\instance.key", :cause=>{:exception=>Java::JavaSecuritySpec::InvalidKeySpecException, :message=>"Neither RSA, DSA nor EC worked"}}
[2022-05-27T15:26:38,281][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: File does not contain valid private key: C:\PATH\logstash-8.0.1\config\certs\instance.key>, :backtrace=>["C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:239:in `new_ssl_handshake_provider'", "C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:193:in `create_server'", "C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:178:in `register'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:390:in `start_inputs'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:315:in `start_workers'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["C:/PATH/logstash-8.0.1/probe-pipeline6.conf"], :thread=>"#<Thread:0x25e6671 run>"}
[2022-05-27T15:26:38,281][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-05-27T15:26:38,296][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-05-27T15:26:44,774][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-05-27T15:26:44,837][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:44,853][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:44,899][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:44,915][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:44,915][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:44,962][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:44,964][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:44,964][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:44,987][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:44,987][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:44,987][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:45,077][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:45,093][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2022-05-27T15:26:45,093][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:45,115][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:45,115][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:45,115][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:45,115][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:45,131][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:45,131][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
[2022-05-27T15:26:45,162][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:xxxxxx@localhost:9200/]}}
[2022-05-27T15:26:45,200][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_internal:xxxxxx@localhost:9200/"}
[2022-05-27T15:26:45,216][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.0.1) {:es_version=>8}
[2022-05-27T15:26:45,216][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-05-27T15:26:45,231][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:45,231][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-05-27T15:26:45,231][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-27T15:26:45,231][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-05-27T15:26:45,231][WARN ][logstash.filters.grok ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-05-27T15:26:45,419][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/PATH/logstash-8.0.1/probe-pipeline6.conf"], :thread=>"#<Thread:0x766f19c3 run>"}
[2022-05-27T15:26:45,688][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.22}
[2022-05-27T15:26:45,735][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-05-27T15:26:45,750][ERROR][logstash.inputs.beats ][main] SSL configuration invalid {:exception=>Java::JavaLang::IllegalArgumentException, :message=>"File does not contain valid private key: C:\\PATHk\\logstash-8.0.1\\config\\certs\\instance.key", :cause=>{:exception=>Java::JavaSecuritySpec::InvalidKeySpecException, :message=>"Neither RSA, DSA nor EC worked"}}
[2022-05-27T15:26:46,240][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: File does not contain valid private key: C:\PATH\logstash-8.0.1\config\certs\instance.key>, :backtrace=>["C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:239:in `new_ssl_handshake_provider'", "C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:193:in `create_server'", "C:/PATH/logstash-8.0.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:178:in `register'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:390:in `start_inputs'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:315:in `start_workers'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "C:/PATH/logstash-8.0.1/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["C:/PATH/logstash-8.0.1/probe-pipeline6.conf"], :thread=>"#<Thread:0x766f19c3 run>"}