When the watch is executed using scheduler, the log display a password for basic authentication (in DEBUG log), which is different from the one when the same watch is executed explicitly using _execute.
I doubt whether decryption is being done correctly when the watch is running in scheduled mode.
Also, based on the logs, I was trying to follow by the actual implementation using xpack code (ExecutionService.java) to see any potential issues, but the WatchExecutionContext referred in the class could not be found anywhere (though its child Manual and Triggred) were found in git. Wondering if it is related to any cacheing on http input, and type of trigger...Should not be though!
This indeed looks as if the system key is not what is expected.
Do you see any chance, you could provide a reproduction? I am extremely keen to take a closer look at this.
When you run the execute watch api, do you specify the password in cleartext or do you also use the ::es_encrypted:: syntax?
I assume you did this, but just to make sure, you are running syskeygen once, and then copy that single file to every node via puppet and call elasticsearch-keystore add-file xpack.watcher.encryption_key - correct?
I missed "core", and was looking inside the same path as Manual and Triggered execution context.
org.elasticsearch.xpack.core.watcher.execution.WatchExecutionContext
When I run execute watch API, I do not provide any password, rather just call
"_xpack/watcher/watch/<>/_execute" with no input
The watch is deployed and is available in .watches indice, along with its schedule.
When the cluster was created, encryption_key settings were created and added to keystore
elasticsearch-keystore add-file xpack.watcher.encryption_key <>
I ensure that both the manual execution and triggered execution did happen from the same node in cluster (the logs are observed from the same node).
ok, all of this looks correct on your side. One last thing: What happens when you run the execute watch API, without specifying the watch but specifying the watch in the body of the API and making sure you are using the ::es_encrypted:: password instead of the plain text one (I hope this explanation/ask makes sense)?
Also what can we do, so that I can try to reproduce this?
If I understand correctly we are trying to check if decryption happens when I execute using my AD on the encrypted text.
It worked when I run watch with watch body using API (I'm using Dev tools of Kibana). Ensured that watch body had the password as "::es_encrypted::Tr+8FDmZ3z.............." as I see in .watches indice.
An wild/absurd guess -- Is it the access to file elasticsearch.keystore causing any issue during scheduled execution. The user context executing the watch not having access to file? Not sure which user context access the file. If this is the case, the log file should show access error in node logs, but it shows security exception when calling input on http. Below is what I see for the file though.
NOTE: The elasticsearch process should be running in elasticsearch user that will have access to the file
-rw-rw---- 1 root elasticsearch 1397 Apr 11 20:38 elasticsearch.keystore
To Reproduce
Two Clusters (C1 has the data, C2 has the watch)
C1 has a index that is read using http input from C2, fail/success on condition and mail
Both C1 and C2 has xPack configured
C1 Access C2 on input using basic authentication, and "elastic" admin user/password
C1 and C2 are accessible only by https (SSL enabled)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
