Security Error on scheduled watches

Elastic Stack Elasticsearch
1

I'm able to sucessfully execute the watch with http input via enforced execution.
POST _xpack/watcher/watch/<>/_execute
NOTE: http input is another Elastic search.

But if the same is scheduled, getting security exception on input against every scheduled run accessing input. It also provides a misleading info on input status as success.

ES Version 6.2.1

"result": {
"execution_time": "2018-04-17T21:41:02.786Z",
"execution_duration": 106,
"input": {
"type": "http",
"status": "success",
"payload": {
"_headers": {
"www-authenticate": [
"Basic realm="security" charset="UTF-8""
],
"content-length": [
"345"
],
"content-type": [
"application/json; charset=UTF-8"
]
},
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "failed to authenticate user [elastic]",
"header": {
"WWW-Authenticate": "Basic realm="security" charset="UTF-8""
}
}
],
"type": "security_exception",
"reason": "failed to authenticate user [elastic]",
"header": {
"WWW-Authenticate": "Basic realm="security" charset="UTF-8""
}
},
"_status_code": 401,
"status": 401
}
Any guidance is helpful.

2

can you share the full watch? can you share the output of the execute watch API?

3

Hi Alex,

The output is exactly as pasted in my original text. It comes under results section. The rest all are inputs. I don't have the complete text of input handy, but it is http input _search ..ing on a indice in elastic with a simple query to check for records. The condition is if hits>0, do the action.

The es is accessed using user/password = basic authentication. Once the watch is created, and executed using xpack _execute, it executes fine. But if the same is executed as part of schedule it fails. If it is a script error, it could be due to script issue, but the error is on security @schedule

Will get the watch script though.

4

I have no doubt about the output you pasted, I just would like the whole watch plus its execution results, which is stored in that response. This eases debugging a lot. Thank you!

5

Sure Alex. Will provide the info once I'm connected back

6

Hi Alex, Below is the watch from .watches indice

{
"_index": ".watches",
"_type": "doc",
"_id": "<>",
"_score": 1,
"_source": {
"trigger": {
"schedule": {
"interval": "15m"
}
},
"input": {
"http": {
"request": {
"scheme": "https",
"host": "<>",
"port": 9200,
"method": "get",
"path": "/filebeat.log.*/_search",
"params": {},
"headers": {},
"auth": {
"basic": {
"username": "elastic",
"password": "::es_encrypted::<<>>"
}
},
"body": """{ "size":0, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now-{{ctx.metadata.interval}}", "lte": "now" } } } ] } } }"""
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"throttle_period_in_millis": 1800000,
"actions": {
"mailaction": {
"email": {
"profile": "standard",
"priority": "high",
"to": [
"{{ctx.metadata.alertto}}"
],
"subject": "Alert - Logstash sending data to filebeat",
"body": {
"html": " Logstash sent {{ctx.payload.hits.total}} entries to filebeat default in last {{ctx.metadata.interval_text}}

"
}
}
}
},
"metadata": {
"interval": "15m",
"alertto": "<>",
"interval_text": "15 minutes"
},
"status": {
"state": {
"active": true,
"timestamp": "2018-04-18T00:13:13.367Z"
},
"actions": {
"mailaction": {
"ack": {
"timestamp": "2018-04-18T00:13:13.367Z",
"state": "awaits_successful_execution"
}
}
},
"headers": {
"_xpack_security_authentication": "<>"
},
"version": -1,
"last_checked": "2018-04-18T18:28:37.613Z",
"execution_state": "execution_not_needed"
}
}
}

The confusing part is that the status as present in status section doesn't co-relate with watch history. The watch history indice cleary indicates the security exception, though POST _xpack/watcher/watch/<>/_execute cleanly executes the watch

Regards
Karthik R

7

One more question just to get a more complete picture. When you run the execute watch API, are you logged in as the same user who put the watch?

8

Hi Alex,

I run the watch using my AD integrated ID (myid).

When the watch is created, it is created with a user that has watcher_admin role, which creates entry in .watches indiceThe user is native and not AD integrated.

The watch by itself access another cluster using http input with appropriate user and password. The uid that we are testing with is admin in the target cluster (basic authentication)

Regards
Karthik R

9

I also observed that the watch fails on inputs that read from the same cluster (other than .watches indice). Should we give any alternative permissions? or execute a watch using specific user id?

Regardless, the http input is explicitly specified with auth settings.

Also, when I deploy the watch with a user who has admin access to the cluster where watches are deployed, it can run on schedule if the indices are read from the same cluster where watch is present. But, still fails on security as detailed above if the data is read using http input with proper security settings.

10

just to take this out of the equation: Can you store the watch as a user that is not in AD and retest? Even though the bug seems to be somewhere else, I just want to reduce the problem scope here.

Also, is it possible that you changed access rights for that user some point after the watch was created (again I do not think that this is the issue, it is just about reducing problem scope).

And one last thing, which may be an issue: Did you upgrade from an earlier Elasticsearch version? I see that the password is encrypted. Did you run bin/elasticsearch-keystore add-file xpack.watcher.encryption_key <filepath>/system_key as mentiond in https://www.elastic.co/guide/en/x-pack/6.2/encrypting-data.html

11

Hi Alex,

We have not upgraded to any newer version, but we did respin some of of our nodes. But, we ensure that all the watches are redeployed whenever we respin any of the nodes.

W.r.t creating watch, below are the scenarios tested

  1. Create watch using admin user,
    Forced Execute - Working (Called REST API using AD user that has admin role)
    Scheduled Execution - Working if the data is from same cluster as watch
    Scheduled Execution - Not Working if the data is from another cluster accessed using http input.
    NOTE: http input is using basic authentication, which is set with admin user of the called cluster

  2. Create watch using user with watcher_admin role
    Forced Execute - Working (Called REST API using AD user has admin role)
    Scheduled Execution - Not Working if the data is from same cluster as watch
    Scheduled Execution - Not Working if the data is from another cluster accessed using http input.
    NOTE: http input is using basic authentication, which is set with admin user of the called cluster

  3. Create watch using AD user which has admin role
    Forced Execute - Working (Called REST API using the same AD user that created watch)
    Scheduled Execution - <<Haven't tested this as this is not the core scenario>>
    Scheduled Execution - Not Working if the data is from another cluster accessed using http input.
    NOTE: http input is using basic authentication, which is set with admin user of the called cluster

Regards
Karthik R

12

Hey,

can you answer also the question regarding the keystore? Did you do this forall the nodes that have been respinned (as I am not sure what this means, just restarting or replacing the whole VM?).

Also can you put a full watch history entry of a failed execution? Scrolling up I only see snippets and I would like to see one full run of possible.

--Alex

13

Hi Alex,

Yes, the scenarios I listed represent the same. The watch is created with a non-AD user and tested. The watches are always created with non-ad user only . I use AD user only when manually enforce watch execution using _execute API.

Also, verified that we have added the encryption file using the command exactly as mentioned in the link referenced.

NOTE: Chef bash settings, that we do to set the file. Also, confirmed by looking into elasticnode checking the presence of file elasticsearch.keystore.

bash "watch encryption" do
user 'root'
group 'elasticsearch'
code <<-EOH
echo "true" | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin xpack.watcher.encrypt_sensitive_data
/usr/share/elasticsearch/bin/elasticsearch-keystore add-file xpack.watcher.encryption_key #{node[:es][:config][:path]}/x-pack/system_key
EOH
end

NOTE:
elasticsearch.keystore is not human readable file, and is expected.

The strange behavior is that the same watch runs when using _execute API explicitly, but not using the schedule.

14

ok, there is one more difference.

When you are using the execute watch API, then you are running the execution on the node, you are hitting it with (which means either the node you specified via curl or the node specified in your kibana configuration).

When the scheduled execution happens, this happens on the same node, where the .watches shard is, you can see which node in the watch history entry.

Can you please rerun the elasticsearch-keystore add-file call on those nodes holding the data and restart them?

I do think that there is a divergence.

The problem is, that it is not really possible to tell, if there is just stuff in the keystore or really the contents of that system key, because all you can do is bin/elasticsearch-keystore list and see if xpack.watcher.encryption_key is set in there.

One thing you can ensure though, is that the md5sum/shasum of that system key is the same on all the nodes, no matter if a datanode or the node you are connecting to with kibana.

15

Thanks Alex.

Yes, the key is found by listing.

./elasticsearch-keystore list
bootstrap.password
keystore.seed
xpack.watcher.encrypt_sensitive_data
xpack.watcher.encryption_key

xpack.watcher.encrypt_sensitive_data is true in Yml as well, but added to keystore because of the chef setting i put earlier. It should not be a issue I believe.

W.r.t Kibana - Haven't specified any specific node, rather it refers to a Load Balancer sitting on top of ES nodes. The watches sit in a cluster only with 3 nodes, and all play master/data.

Will try to re-add the file though, and get back on watch history node reference.

16

Hi Alex,

Also validate that the node where watch was executing (as observed from watch history) do have the xpack.watcher.encryption_key listed.

17

Have you tried using curl to execute the watch manually on all of your nodes? Does that work as well?

18

Hi Alex,

Yes. It works fine with individual es nodes, where watch shard is present.

19

I was not only talking about where the watch shard is, but on any node in the cluster. Is this also the case?

I am slowly running out of ideas, if the above works.

Do you see any chance of reproduction outside of that particular system? Does the same happen if you set up a new completely independent one node cluster?

20

Hi Alex,

I have a dev cluster, where the issue is same.

I'm not clear on the point "but on any node in the cluster". Could you please add more details.

I executed the watch from each node where a primary or replica of .watch indice being present to confirm that issue persist when executing at node directly.

Sorry - I should take back my previous statement "Yes. It works fine with individual es nodes, where watch shard is present.".. Looks like I missed "doesn't".

Even if I run the watch from individual nodes, I get security exception.