Does http_poller handle https?

Elastic Stack Logstash
1

This input plugin looks perfect for what I need except I need to call https and so must provide certificate information. I found this closed github issue, https://github.com/logstash-plugins/logstash-input-http_poller/issues/2, and I followed the links and read the commits but see nothing about https. I thought about opening another github issue but thought I'd better post here first in case someone has better insight.

Thanks,
Craig

2

It does, check the docs - https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html

3

Thanks for the link Mark. I'm getting a little further. I'm specifying "truststore", "truststore_password", "client_cert" as myCert.cer, and "client_key" as myKey.key. I'm still getting a "_http_request_failure" though and I'm having trouble determining why.

I'm getting "error" => "string not matched" and "backtrace" => [
"org/jruby/rubyString.java:3912:in []='", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/event.rb:73:ininitialize'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-1.0.1/lib/logstash/codecs/json.rb:44:in decode'", file:/opt/logstash/vendor/bundle/jruby/1.9/lib/jruby.jar!/jruby/java.util.rb:7:ineach'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-1.0.1/lib/logstash/codecs/json.rb:44:in decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-1.1.2/lib/logstash/inputs/http_poller.rb:156:in 'handle_success'", ... "Manticore$$Response_1817397330.gen:13:incall'"
]

I'm guessing that being in handle_success is a good thing but I have no idea what the string comparison is about. Any ideas? It should be returning something like ["a","b","c"]. Am I correct in passing in .cer and .key files? Are there any https examples I can look at?

Thanks,
Craig

4

UPDATE: I changed the url I was using to one that returns a json object rather than an array. Now I'm getting a different response: "403 forbidden, unable to authenticate user - no certificate provided". I'm sure my .cer and .key files are there and are valid so I assume I should be passing in something else. A https example would go a long way right now :slight_smile: Again, any help greatly appreciated.

Craig

5

UPDATE: I changed the .cer to a .crt file with the start/end certificate string from my pem file. Same result.
I see in the rest service's code though that it expects a request attribute named "javax.servlet.request.X509certificate" for the cert and it apparently isn't finding it. Does this plugin pass my .crt in as that?

p.s. if I'm creating too much noise or doing anything wrong by posting here, please let me know. I'm in a bit of a vacuum here.

6

UPDATE: I'm supplying a .crt and a .key file but I see no way to supply the key password...!?
Examples people! We need examples!

7

Craig, you'll need to set the 'cacert' option. truststore is for java .jks keystore files.

8

Sorry I don't understand. I have the "truststore" set to my truststore.jks and "truststore_password" set to its password. That's for "who I trust". That much is correct, yes? As for "who I am", I've tried not setting "client_cert" and "client_key" and instead setting "cacert" to different formats of my keystore: a .pem, a .cer and a .jks. What should it be set to? Also I see no way to supply the keystore password. Thanks for replying by the way :smile:

9

CraigFoote can you open an issue for providing the keystore_password option? That was an oversight of mine in developing the API. Currently there's no way to provide one.

You can of course use PEM files for now. For "who I am" client_cert / client_key should work. That they aren't is a bug unfortunately. Are you sure the client isn't connecting those values?

10

Just so I create the ticket right, am I correct in understanding the intent of the 'cacert' option as being the certificate keystore, e.g. 'myCert.jks'? And that normally a password is required so we're missing a 'cacert_password' option?

Sorry I'm being a little dense but I still don't fully understand. Am I correct in setting the 'truststore' and 'truststore_password' options? And, until the 'cacert_password' option is available, to what should I set the 'client_cert' and 'client_key' to exactly? My understanding of a .pem file is that it contains public certificates and optionally private keys, the latter requiring a password. I think I need to supply my private key and somehow its password but I'm new to this stuff and it seems you know how it can work.

Oh I just remembered, I checked the code of the rest service I'm calling and it expects a request attribute called 'javax.servlet.request.X509Certificate'. Do you know if this is being set?

11

bumping

12

Bumping again. Sorry I really need an answer on this.

13

Bumping again. Sorry I really need an answer on this.

14

Bumping again. Sorry I really need an answer on this. Is this pissing anyone off as much as me yet?

15

Craig, cacert is not for JKS. cacert is for .pem files . If you try again with that option it should work for you.

16

It looks like you may be able to convert between the formats using the instructions here, though I haven't tried it myself. http://um.terracotta.org/index.html#page/Universal_Messaging_Documentation_Set/to-jksToPem.html

17

Thanks for responding Andrew. I tried setting the 'cacert' option to my non-password-encoded key.pem file that I created using the link you provided. Unfortunately I'm getting:

"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

Any ideas?

18

Do I need to use the truststore and truststore_password options. I did and got a different error: "key usage does not allow key encipherment". I don't know if I'm getting further or if providing the truststore options are causing a different error. As the truststore is "who I trust". it seems to me I shouldn't need to provide it; I should only need to provide "who I am" via the private key, right?

19

Apologies Craig, I missed that you were using a clientside KEY.

I think I can clear this all up now.

'cacert' lets you set an alternate CA, performing the same role as a truststore. This lets you accept a custom cert from a remote server.

Specifying a custom clientside certificate for x509 auth requires setting 'client_key' to the local key for your client as well as setting 'client_cert' with the signature for that key.

I apologize for the confusion. SSL is generally very confusing.

1 Like
20

Did you ever this issue resolved? I'm getting the same error trying using http_poller plugin on https server.

my config is below
input {
http_poller {
urls => {
test => {
method => get
url => "https://labhub01.prolab.inc/lwl/api?json={"inspector"%3A"4"%2C"basis"%3A"machines"%2C"date"%3A"2%20hours"%2C"limit"%3A"0"}"
verify_cert => false
cacert => "c:\temp\cacert.pem"
}