Security Error on scheduled watches

Hi Alex,

After enabling debug log, I observe below

When the watch is executed using scheduler, the log display a password for basic authentication (in DEBUG log), which is different from the one when the same watch is executed explicitly using _execute.

I doubt whether decryption is being done correctly when the watch is running in scheduled mode.

[2018-05-02T19:23:12,814][DEBUG][o.a.h.headers ] http-outgoing-498 >> Authorization: Basic <>
[2018-05-02T19:23:12,926][DEBUG][o.a.h.headers ] http-outgoing-498 << HTTP/1.1 401 Unauthorized

But not sure whether this is intended

Also, based on the logs, I was trying to follow by the actual implementation using xpack code (ExecutionService.java) to see any potential issues, but the WatchExecutionContext referred in the class could not be found anywhere (though its child Manual and Triggred) were found in git. Wondering if it is related to any cacheing on http input, and type of trigger...Should not be though!

This indeed looks as if the system key is not what is expected.

Do you see any chance, you could provide a reproduction? I am extremely keen to take a closer look at this.

When you run the execute watch api, do you specify the password in cleartext or do you also use the ::es_encrypted:: syntax?

I assume you did this, but just to make sure, you are running syskeygen once, and then copy that single file to every node via puppet and call elasticsearch-keystore add-file xpack.watcher.encryption_key - correct?

FYI, here is the link to the WatchExecutionContext

Thanks Alex.

I missed "core", and was looking inside the same path as Manual and Triggered execution context.
org.elasticsearch.xpack.core.watcher.execution.WatchExecutionContext

When I run execute watch API, I do not provide any password, rather just call
"_xpack/watcher/watch/<>/_execute" with no input

The watch is deployed and is available in .watches indice, along with its schedule.

When the cluster was created, encryption_key settings were created and added to keystore
elasticsearch-keystore add-file xpack.watcher.encryption_key <>

I ensure that both the manual execution and triggered execution did happen from the same node in cluster (the logs are observed from the same node).

ok, all of this looks correct on your side. One last thing: What happens when you run the execute watch API, without specifying the watch but specifying the watch in the body of the API and making sure you are using the ::es_encrypted:: password instead of the plain text one (I hope this explanation/ask makes sense)?

Also what can we do, so that I can try to reproduce this?

Hi Alex,

Thanks.

If I understand correctly we are trying to check if decryption happens when I execute using my AD on the encrypted text.

It worked when I run watch with watch body using API (I'm using Dev tools of Kibana). Ensured that watch body had the password as "::es_encrypted::Tr+8FDmZ3z.............." as I see in .watches indice.

An wild/absurd guess -- Is it the access to file elasticsearch.keystore causing any issue during scheduled execution. The user context executing the watch not having access to file? Not sure which user context access the file. If this is the case, the log file should show access error in node logs, but it shows security exception when calling input on http. Below is what I see for the file though.
NOTE: The elasticsearch process should be running in elasticsearch user that will have access to the file
-rw-rw---- 1 root elasticsearch 1397 Apr 11 20:38 elasticsearch.keystore

To Reproduce

  1. Two Clusters (C1 has the data, C2 has the watch)
  2. C1 has a index that is read using http input from C2, fail/success on condition and mail
  3. Both C1 and C2 has xPack configured
  4. C1 Access C2 on input using basic authentication, and "elastic" admin user/password
  5. C1 and C2 are accessible only by https (SSL enabled)
  6. elasticsearch-keystore add-file xpack.watcher.encryption_key <> (@C2)
  7. xpack.watcher.encrypt_sensitive_data to true (@C2)
  8. Both are AD integrated
  9. Execute the watch using schedule to fail
  10. Execute the watch using API to succeed

xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.ssl.key: ..../es.key
xpack.ssl.certificate: .../es.crt
xpack.ssl.certificate_authorities: [ "XXXX.crt", "YYYYY.crt" ]
Xpack Order for AD integration
active_directory:
type: active_directory
order: 0
domain_name: .....
url: ldap://.......
unmapped_groups_as_roles: false
native1:
type: native
order: 1

NOTE: Masked certain values with XXX,YYYY,.... etc.

I think I found a way of reproducing this, seems not to be bound to LDAP/AD. Thanks for all your input.

Ok, I found the issue. It is basically a double encryption of the secret, when the watch is executed on a scheduled base.

This is already fixed in the 6.3 branch.

Thanks for all the time helping out here, it's much appreciated!

1 Like

Thanks Alex for the great support here.

Double encryption is interesting :slight_smile:

Looking forward for 6.3 official release.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.