Watcher webhook action to Zenoss - PKI authentication?

Elastic Stack Elasticsearch
1

I am (attempting) to configure watcher with a webhook action to generate an event in ZenOSS when a specific pattern is found in an index.
The ZenOSS instance is configured for LDAP authentication.

Is it possible to configure watcher to use PKI authentication to avoid needing to put a username/password in the watch?
I've read that the credentials could be encrypted using shield with "watcher.shield.encrypt_sensitive_data: true" but I would prefer to use keys if possible.
I've also read that "You can also use PKI-based authentication when submitting requests to a cluster secured with Shield." but I'm not sure if the same applies to a request sent elsewhere.

2

Hey,

check out the docs about the webhook action, as it mentions the option to set up the watcher http client for PKI Auth by setting watcher.http.ssl.keystore.path.

Hope this helps!

--Alex

3

@spinscale - Thanks. I'll have a go. I have read through the documentation but I didn't know if it would work as the document says "You can also use PKI-based authentication when submitting requests to a cluster secured with Shield.". I would be using an action that submitted a request to another site.

4

Hey,

shield should be just considered an example here, as you basically just configure a keystore/truststore in java.

--Alex

5

Excelllent - I'll give it a go. Thank you.

Nathan

6

Ok I've been battling this for a while with no success - Any suggestions of where to go next would be very welcome!

We've decided to use basic authentication sent over https. I have imported our root CA certificate, and using a java tool called "SSLPoke" I can get a successful connection from the ES server, to the ZenOSS server using the options -Djavax.net.ssl.keyStore={/path/to/keystore.jks} -Djavax.net.ssl.keyStorePassword={keystore password}

So that seems to suggest that the keystore and certificate configuration is ok.

Our elasticsearch.yml file contains the following section which was added when Shield was configured:
shield.ssl.keystore.path: /path/to/keystore.jks
shield.ssl.keystore.password: **********
shield.transport.ssl: true
shield.http.ssl: true
shield.ssl.ciphers: [ "TLS_RSA_WITH_AES_128_CBC_SHA256", "TLS_RSA_WITH_AES_128_CBC_SHA" ]
shield.ssl.protocol: TLS
hostname_verification: false

And the following section added for Watcher:
watcher.shield.encrypt_sensitive_data: true
watcher.http.ssl.keystore.path: /path/to/keystore.jks
watcher.http.ssl.keystore.password: ********

And these lines added for troubleshooting
watcher.http.ssl.truststore.path: /path/to/keystore.jks
watcher.http.ssl.truststore.password: ********
shield.http.ssl.truststore.path: /path/to/keystore.jks
shield.http.ssl.truststore.password: ********

I am still getting the following error when a watch executes:
{
"id": "zenoss_webhook",
"type": "webhook",
"status": "failure",
"reason": "SSLHandshakeException[sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]; nested: ValidatorException[PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]; nested: SunCertPathBuilderException[unable to find valid certification path to requested target]; "
}

The action section of the watch looks like this:
"actions" : {
"zenoss_webhook" : {
"throttle_period" : "2m",
"webhook" : {
"scheme" : "https",
"host" : "zenoss-server.our.domain",
"port" : 443,
"method" : "post",
"path" : "/zport/dmd/evconsole_router",
"params" : { },
"headers" : {
"Content-Type" : "application/json"
},
"auth" : {
"basic" : {
"username" : "zenoss user name here",
"password" : "::es_encrypted::*********"
}
},
"body" : "{"action":"EventsRouter","method":"add_event","data": [{"summary":"{{ctx.payload.hits.hits.0._source.message}}","device":"{{ctx.payload.hits.hits.0._source.host}}","component":"","severity":"Critical","evclasskey":"","evclass":"/App"}],"type":"rpc","tid":1}"
}
}
}

We do have a platinum license so I think it may be time to raise a support ticket unless anyone can spot anything I've missed.

7

I thought I should return and post an update. We now have this working. The problem was down to me not understanding which certificates to place where.
The working config has:
ES/Shield keystore contains the root CA certificate from our IDM server, and the host certificate for that server (signed by our CA)
The Zenoss server has the host certificate for itself, also signed by our CA.

It sounds simple now but I was missing the host certificate for the ES server in the shield keystore.

8

Hey nathan,

glad you sorted it out!

is there anything we can do on our documentation side to improve this and not have others run into this in the future? happy to update docs or whatever is needed...

--Alex

9

In all honesty, it's probably down to me not properly understanding how SSL works!
:blush: