We are finding that the standalone logstash does not connect over to the ece build on docker. We are getting certificate errors. Is that because of stunnel handling the TLS activity or is there something else that we are missing?
Thanks,
Linda
Can you provide any more details on what your config looks like, or what errors exactly you're getting
A common problem: by default ECE ships with self-signed certificates, so to connect from logstash you either need to turn SSL verification mode off (maybe https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_certificate_verification?), or upload certs (https://www.elastic.co/guide/en/cloud-enterprise/2.4/ece-manage-certificates.html)
The error is that the certificate is not recognized. we are in the middle of a lab rebuild or I could have grabbed the error. I will try the ssl verification mode - I had heard of that but not yet tried it. As a side note - I do need to update the logstash yml still don't I or can I just point to the server in the logstash.conf for the output
output {
elasticsearch {
hosts => ["http://elasticsearch docker container info:port"]
username= 'elastic'
password ='xlfksfljeoiroeri.sample'
index => "linda_sample"
document_type => "Sample1_20_20"
}
stdout {codec => rubydebug }
}
It looked like you could just put the SSL verification setting in the config , but I'm definitely not an expert! The logstash forum contains folks who know more about this sort of thing 
(eg a generic approach that handles self-signed certs should work with ECE - the only ECE wrinkle is that you can't get hold of the signing CA, it's internal to the platform - so you have to address it client side by making SSL validation less strict
In general I'd recommend getting hold of a publicly signed cert an uploading to ECE, so you don't have to go through this exercise with every new client with strict SSL validation
Alex - thanks very much - we have tried that but seem to still be getting errors that I have been told are due to the certificate. For your historical information - The verified/ssl certificate was uploaded to the Cloud UI of the ECE 2.4.3 implementation and then a copy of that certificate was put on the Logstash server. We then put that entry into the logstash configuration 
We need to have secured communication so less strict is not really an option.
elasticsearch {
hosts => ["https://c8e98726538340d892c810ed49c9ef6e.1xx.xxx.xxx.ip.es.io:9243"]
user => "elastic"
password => "IaS4q4S0tBoUxSAMPLE"
ssl => true
(we also tried to add the ssl_certification_verification value as well - same errors)
cacert => "/usr/share/logstash/Full_new.pem"
index => "sample_Index_ece"
}
stdout{}
}
We then try to run with the normal bin/logstash -f filename command but are not getting past the following errors.
An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in block in validate_all'", "org/jruby/RubyHash.java:1428:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:284:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in '"]}
[2020-03-13T16:02:36,171][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Has anyone seen this and can they point us in the right direction?
Appreciate your help.
Linda
Did you say someone thought that ArgumentError: Path "/var/lib/logstash" must be a writable directory. It is not writable. was related to the certificate? I'm not a logstash expert, but it looks like an unrelated config error?
I agree with you - I believe that there are two different issues. Is there someone who could work with us to go through the logs? I can send the logs over and go through all the steps that we have performed and the testing that we have done. We have a large implementation of ECE 2.4.3 but need to include the logstash on a separate vm with the indexes being created over in the docker containers. Your help or direction would be great! Thanks for the quick response.
The error you currently have seems super standard, eg Discuss link
Once you get past that, your cacert idea seems like it should work, but of course certificate validation can be slightly tricky. Probably the logstash forum is best placed to debug the specific errors that logstash is spitting out
I did fix the permissions error with the sudo chown -recursive /var/log/logstash entry. But I am still getting this error. Should I copy this whole thread over to there or can you forward it internally? Again - thanks for your help and quick reply.
I can move it for you, but empirically I believe you will get a better response if you start a new thread just focussing on the error
Incidentally chown might not help depending on what the permissions are? Can you do a ls -la on /var/lib/logstash? You'd want u+w as well as it being owned by the right user
(I'd be pretty surprised if you got an error saying Path "/var/lib/logstash" must be a writable directory. It is not writable if it was in fact writable )
Alex -
I did create a new topic and put all the history into that I have share with you..
Here is the command ls -la on the /var/lib/logstash directory.. what am I missing. Does the .lock or the uuid need to have more rights?
cd /var/lib/logstash
[ece_admin@dev-logstash-01 logstash]$ ls -la
total 8
drwxrwxr-x. 4 logstash logstash 69 Mar 4 14:15 .
drwxr-xr-x. 63 root root 4096 Mar 4 14:14 ..
drwxr-xr-x. 2 logstash logstash 6 Mar 4 14:15 dead_letter_queue
-rw-r--r--. 1 logstash logstash 0 Mar 4 14:15 .lock
drwxr-xr-x. 2 logstash logstash 6 Mar 4 14:15 queue
-rw-r--r--. 1 logstash logstash 36 Mar 4 14:15 uuid
Thanks again.
I can't see anything wrong with those permissions, and in fact we can see that the dir is now writable, because it's created all those extra files and folders
Are you sure it's still giving the same non writable permission error?!
(btw that logstash thread is a bit noisy to get traction ... my suggestion would be simply to include the current config, the permissions, and the error you are seeing)
So the "needs to be writeable" is gone - now we have that the elasticsearch is not reachable.
[Ruby-0-Thread-5: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@sample docker.xxx.xxx.xxx.xxx.ip.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@sample.xxx.xxx.xxx.xx.ip.es.io:9243/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}******
I removed the Cloud.ID and Cloud.auth from the logstash.yml earlier today and was attempting to put back elements and see what would fix the issues.
I put the ssl_certificate_verification =>true back but did not put the path in yet. Should I try that next or is it better to do it within the logstash.yml.?
appreciate your help
Linda
I don 't know if one location is preferable over the other - I believe either is fine
That log message together with how you built the certificate that is loaded is the key information that logstash/SSL experts should be able to help with
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.