# ES Node Disconnects after enablign Shield

**URL:** <https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [October 25, 2016, 6:10pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917 "2016-10-25T18:10:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [October 25, 2016, 6:10pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/1 "2016-10-25T18:10:20Z")

</div>

Hi Team,  
Not sure where this topic belongs Shield or Elasticsearch.

We have ELK up and running (1 master, 1 client and 4 data nodes) and now we implemented Shield (Active Directory), this is also working fine as far as user authentication and authorization is concern.

But, i am continuously facing a problem, Kibana is getting timedout. And reason being Elasticsearch cluster becomes RED. Then it auto recovers and becomes GREEN. It's going on regularly, obviously i can query only when ES is not RED.

ES Logs says node discovery is getting timed out.

PFB elasticsearch discovery settings, rest settings are default:

# --------------------------------- Discovery ----------------------------------

discovery.zen.ping.multicast.enabled: true  
discovery.zen.ping.unicast.hosts: ["_._.1.55","_._.1.254","_._.11.99", "_._.11.98","_._.1.134","_._.1.50"]

ES Master node log message is a

 ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9d4ec652626d866b1a0c936ef8b01fc130d98e1e.JPG)ttached.

Another issue is: "failed to execute bulk item (index) index" where as my logstash role has all access:

POST /\_shield/role/logstash  
{  
"cluster": ["all"],  
"indices": [  
{  
"names": ["_"],  
"privileges": ["_"]  
}  
]  
}

Thanks & Regards

---

<div class="post-metadata">

**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\
**Post date:** [October 25, 2016, 6:54pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/2 "2016-10-25T18:54:10Z")

</div>

Please don't use screenshots for logs; it is really hard to parse. I'd recommend putting logs in a gist or using pastebin. It looks like you are hitting timeouts on node stats actions. Are your nodes overloaded? Do you have GC issues or anything like that.

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [October 25, 2016, 8:10pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/3 "2016-10-25T20:10:10Z")

</div>

I got a 5000 word limit restriction so i quickly uploaded snapshot.

Nodes may be overloaded due to data volume, what can be done here? But node's CPU (\< 20) and Mem (\<60) utilization is low, adding more node will help?

I don't think it's GC issue without Shiled everything works perfectly. I am suspecting, nodes are pinging each other to collect status and this activity is getting delayed because of shield, some authorization delay may be?

I validate system\_key and also enabled shield audit, it doesn't say any issue.

[2016-10-25 13:03:08,791] [es-master-node] [transport] [access\_granted] origin\_type=[local\_node], origin\_address=[_._.1.55], principal=[\_\_marvel\_user], action=[cluster:monitor/nodes/stats]  
[2016-10-25 13:03:08,791] [es-master-node] [transport] [access\_granted] origin\_type=[local\_node], origin\_address=[_._.1.55], principal=[\_\_marvel\_user], action=[cluster:monitor/nodes/stats[n]]  
[2016-10-25 13:03:08,792] [es-master-node] [transport] [access\_granted] origin\_type=[local\_node], origin\_address=[_._.1.55], principal=[\_\_marvel\_user], action=[indices:data/write/bulk]  
[2016-10-25 13:03:08,793] [es-master-node] [transport] [access\_granted] origin\_type=[local\_node], origin\_address=[_._.1.55], principal=[\_\_marvel\_user], action=[indices:data/write/bulk[s]], indices=[.marvel-es-data-1,.marvel-es-data-1,.marvel-es-data-1,.marvel-es-data-1,.marvel-es-data-1]  
[2016-10-25 13:03:09,033] [es-master-node] [transport] [tampered\_request] origin\_type=[transport], origin\_address=[_._.11.99], action=[internal:discovery/zen/unicast]  
[2016-10-25 13:03:09,119] [es-master-node] [rest] [anonymous\_access\_denied] origin\_address=[_._.1.71], uri=[/\_bulk]  
[2016-10-25 13:03:09,409] [es-master-node] [transport] [access\_granted] origin\_type=[transport], origin\_address=[_._.1.254], principal=[kibana-admin], action=[cluster:monitor/health], indices=[.kibana]  
[2016-10-25 13:03:09,411] [es-master-node] [transport] [access\_granted] origin\_type=[transport], origin\_address=[_._.1.254], principal=[kibana-admin], action=[cluster:monitor/health], indices=[.kibana]  
[2016-10-25 13:03:09,570] [es-master-node] [transport] [tampered\_request] origin\_type=[transport], origin\_address=[_._.11.98], action=[internal:discovery/zen/unicast]  
[2016-10-25 13:03:10,532] [es-master-node] [transport] [tampered\_request] origin\_type=[transport], origin\_address=[_._.11.99], action=[internal:discovery/zen/unicast]  
[2016-10-25 13:03:11,059] [es-master-node] [rest] [anonymous\_access\_denied] origin\_address=[_._.1.71], uri=[/\_nodes/http]  
[2016-10-25 13:03:11,071] [es-master-node] [transport] [tampered\_request] origin\_type=[transport], origin\_address=[_._.11.98], action=[internal:discovery/zen/unicast]  
[2016-10-25 13:03:11,130] [es-master-node] [rest] [anonymous\_access\_denied] origin\_address=[_._.1.71], uri=[/\_bulk]

Thanks & Regards,

---

<div class="post-metadata">

**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\
**Post date:** [October 25, 2016, 8:25pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/4 "2016-10-25T20:25:41Z")

</div>

System key is the same on all of the nodes? It doesn't look like it is. The tampered request messages are indicative of a bad signature/system key mismatch:

```
[2016-10-25 13:03:09,033] [es-master-node] [transport] [tampered_request] origin_type=[transport], origin_address=[..11.99], action=[internal:discovery/zen/unicast]
```

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [October 25, 2016, 11:18pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/5 "2016-10-25T23:18:58Z")

</div>

🙂

Yes, system key is same on all nodes but i forgot to change file permission on two of the nodes. Still i am facing timeout issue on Kibana but i believe it's due to ES performance not shiled as no such error. I will debug that more.

Thanks Jay...

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [October 26, 2016, 5:42pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/6 "2016-10-26T17:42:36Z")

</div>

Hi Jay,  
Kibana is still getting timed out, please suggest why "cluster:monitor/nodes/stats[n]" is getting timed out? After shield implementation elaticsearch is using which role for internal node stats?

I am using Active directory integration and created user/role for Logstash and Kibana but i have not done anything for ES yet, no user created with esuser.

PFB log from ES-Master node:  
[2016-10-26 10:21:48,645][INFO][cluster.metadata] [es-master-node] [myproject-dummy-json-log-2016.10.26] creating index, cause [auto(bulk api)], templates [], shards [2]/[1], mappings [json-log]  
[2016-10-26 10:21:48,789][INFO][cluster.routing.allocation] [es-master-node] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[myproject-dummy-json-log-2016.10.26][0], [myproject-dummy-json-log-2016.10.26][1]] ...]).  
[2016-10-26 10:21:48,953][INFO][cluster.routing.allocation] [es-master-node] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[myproject-dummy-json-log-2016.10.26][1], [myproject-dummy-json-log-2016.10.26][0]] ...]).  
[2016-10-26 10:21:49,102][INFO][cluster.metadata] [es-master-node] [myproject-dummy-json-log-2016.10.26] update\_mapping [json-log]  
[2016-10-26 10:21:49,295][INFO][cluster.metadata] [es-master-node] [filebeat-2016.10.26] create\_mapping [json-log]  
[2016-10-26 10:23:07,688][DEBUG][action.admin.cluster.node.stats] [es-master-node] failed to execute on node [hNXte3tURoa7houS\_YjYRw]  
ReceiveTimeoutTransportException[[es-data-node-4][_._.11.98:9300][cluster:monitor/nodes/stats[n]] request\_id [628949] timed out after [15000ms]]  
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:745)  
[2016-10-26 10:23:21,715][WARN][shield.transport] [es-master-node] Received response for a request that has timed out, sent [29027ms] ago, timed out [14027ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es-data-node-4}{hNXte3tURoa7houS\_YjYRw}{_._.11.98}{_._.11.98:9300}{master=false}], id [628949]

Thanks & Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:41pm UTC](https://discuss.elastic.co/t/es-node-disconnects-after-enablign-shield/63917/7 "2017-07-06T13:41:31Z")

</div>