TLS config issues?

Elastic Stack Logstash
1

I'm trying to setup tls on ELK. I have tls and https working between elastic and kibana but no luck with logstash. Below are the logstash configs. I've followed this guide almost to the T till step 7 besides swapping dns for ips in the certificate generation process.

I know logstash/ is not configured properly because I can send data from winogbeats to elastic without issue. But when I try to add ingest logs via logstash whether its a csv or beats agent it fails.

The stack worked as expected prior to adding users and certs. Below is the config I have for the yml file and pipe file. Any ideas?

Logstash.yml

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_writer
xpack.monitoring.elasticsearch.password: passwordhere
xpack.monitoring.elasticsearch.hosts: [ 'https://192.168.1.10:9200' ]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt

01.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/logstash/config/certs/logstash.pkcs8.key'
    ssl_certificate => '/etc/logstash/config/certs/logstash.crt'

  }
}
output {
  elasticsearch {
    hosts => "https:192.168.1.10:9200"
    cacert => '/etc/logstash/config/certs/ca.crt'
    user => 'logstash_writer'
    password => 'passwordhere'
  }
}
2

Do you get error messages?

Does this thread help?

3

That thread sort of helps, but I dont believe that is the issue as when i created the certs I used ip fields instead of hostnames as I've already ran into that issue previously.

Below is a large chunk of the log file. Ignore the errors about cert verification.

:xxxxxx@192.168.1.10:9200/]}}
[2020-10-28T19:04:56,656][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 +indy +jit [linux-x86_64]"}
[2020-10-28T19:04:57,150][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-28T19:04:58,961][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2020-10-28T19:04:58,967][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2020-10-28T19:05:00,731][INFO ][org.reflections.Reflections] Reflections took 40 ms to scan 1 urls, producing 22 keys and 45 values 
[2020-10-28T19:05:00,942][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) URI is not valid - host is not specified", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:82)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
[2020-10-28T19:05:01,009][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2020-10-28T19:05:01,049][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@192.168.1.10:9200/]}}
[2020-10-28T19:05:01,075][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:370:in `block in converge_state'"]}
[2020-10-28T19:05:01,098][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-10-28T19:05:01,104][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@192.168.1.10:9200/"}
4

Here is another error Ive ran into but havent found anything useful online.

[2020-10-28T19:05:01,075][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:370:in `block in converge_state'"]}
[2020-10-28T19:05:01,098][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
5

Does the error go away if you include the // ?

For the other error try enabling log.level debug. A bug was introduced a couple of versions back where the error message is logged at debug instead of warn or error.

1 Like
6

So unfortunately that didnt get rid of the errors (stupid me forgot had a easy typo).
There are still errors which seem to loop, good to know about the error switching mentioned above.

I can't post the entire log on here so here is the pastebin link https://pastebin.com/YiBCAZwJ.
Much appreciated for the help Badger!

7

The fatal error is

[2020-10-28T19:45:27,978][FATAL][logstash.runner          ] An unexpected error occurred! {
:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, 
:backtrace=>[...elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", 
...elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", 
...
...elasticsearch/http_client.rb:341:in `exists?'", 
...elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", 
...elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", 
...elasticsearch/ilm.rb:10:in `setup_ilm'", 
...elasticsearch/common.rb:50:in `block in setup_after_successful_connection'"]}

So it is able to connect to elasticsearch but when it tries to setup ilm it sends a request to elasticsearch to determine whether the rollover index exists. That is getting an error. The elasticsearch logs may give a better error message to explain why it is rejecting the request.

My guess would be a 403 (possibly your user has the wrong role).

It really sucks that logstash.runner does not the additional detail that is attached to the exception (i.e. the http status).

8

So I have a feeling it may not be completely permissions related. I gave the user as much permissions as possible for testing and its still appears to be having issues. Regarding elastic logs, I watched the elastic log as I restarted the logstash service and nothing was written by elastic. Is their another file besides /var/log/elasticsearch/elasticseartch.log i should check? I did not see one when I was checking

I did add a index entry to logstash conf.d entry to see if that helped as I noticed I was missing that
( index => "%{[@metadata][beat]}-%{[@metadata][version]}" )

[2020-10-29T13:46:23,478][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-10-29T13:46:23,583][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:50:in `block in setup_after_successful_connection'"]}
9

Does below error mean anything useful? So far all my research has let to dead posts or posts that tell the user to add something that I already have added to the configs

[2020-10-30T15:27:15,417][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
10

That message is not informative by itself. If you enable log.level debug you may get a better error message.

1 Like
11

Hi Badger, I just wanted to say thank you for your help. While you did not give me the answer your help lead me to it.

Somehow the file permissions for one private key was different and logstash service didnt have access to the key but could access the others. Turning on log debugging gave me a better error code.

Now to tackle winlogbeats to logsatash with tls on.
Cheers,

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.