ES Cloud AWS error 2.2.1

DMNSteve · March 24, 2016, 10:11am

Hi,

Trying to setup a ES cluster in AWS. Managed to succeed and get discovery working with v1.7.

Ive now upgraded to v2.2.1 and im getting the following error

[2016-03-24 09:28:17,234][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport]
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:474)
at java.security.AccessController.checkPermission(AccessController.java:685)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413)
at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361)
at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111)
at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:50)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355)
at com.amazonaws.metrics.AwsSdkMetrics.(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:899)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Any ideas?

ywelsch · March 24, 2016, 10:26am

This might be a bug in the ES AWS plugin. I've asked the plugin developer to have a look.

In the mean time, can you try adding the following permission to your java.policy and check if it works?

permission javax.management.MBeanServerPermission "createMBeanServer";

(to find java.policy see section "Customising the classloader whitelist" in
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-security.html )

DMNSteve · March 24, 2016, 10:56am

Hi Yannick,

Thanks for this, i added the following

permission javax.management.MBeanServerPermission "createMBeanServer";
permission javax.management.MBeanServerPermission "findMBeanServer";

and recieved a new error

[2016-03-24 10:54:28,201][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport]
java.security.AccessControlException: access denied ("javax.management.MBeanPermission" "com.amazonaws.metrics.MetricAdmin#-[com.amazonaws.management:type=AwsSdkMetrics]" "registerMBean")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:474)
at java.security.AccessController.checkPermission(AccessController.java:685)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1830)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:321)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:52)
at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27)
at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355)
at com.amazonaws.metrics.AwsSdkMetrics.(AwsSdkMetrics.java:316)
at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563)
at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504)
at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496)
at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215)
at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240)
at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106)
at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84)
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:899)
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75)
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

any further ideas?

dadoonet · March 24, 2016, 11:07am

Could you open an issue with all those details please?

Thanks for reporting!

DMNSteve · March 24, 2016, 11:15am

No worries at all!

ywelsch · March 24, 2016, 11:33am

Can you add some more permissions?

permission javax.management.MBeanPermission "com.amazonaws.metrics.*", "*";

Thanks a lot for helping us figure this out.

DMNSteve · March 24, 2016, 2:32pm

permission javax.management.MBeanServerPermission "createMBeanServer";
permission javax.management.MBeanServerPermission "findMBeanServer";
permission javax.management.MBeanPermission "com.amazonaws.metrics.*", "*";
permission javax.management.MBeanTrustPermission "register";

Those 4 have gotten rid of the fatals in the logs. But still cant get my 2 nodes talking.

Thanks

DMNSteve · March 24, 2016, 2:41pm

This is my elasticsearch.yml

cloud:
  aws:
    region: eu-west-1
cluster:
  name: elasticsearch
discovery:
  ec2:
    availability_zones: eu-west-1a, eu-west-1b
    groups: security-group-elasticsearch
    host_type: private_ip
    ping_timeout: 30s
  type: ec2
network:
  port: 9200
node:
  name: elasticsearch-2
path:
  data: /usr/share/elasticsearch/data/elasticsearch-2
  logs: /var/log/elasticsearch/elasticsearch-2

I have setup the IAMs roles as described in the cloud aws documentation

ywelsch · March 24, 2016, 4:58pm

Settings look ok to me.

Can you set the log-level to DEBUG for discovery packages and see if anything particular stands out?

org.elasticsearch.discovery.ec2
org.elasticsearch.discovery.zen

We will do testing on our end as well.

Andrew_Stangl · March 30, 2016, 10:42am

It's worth noting that ES 2.x defaults to binding only to localhost.

I was experiencing the same issue, and adding:

network.host: 0.0.0.0

to elasticsearch.yml, fixed the issue for me .. hope that helps someone

ywelsch · April 12, 2016, 2:57pm

As I've written on the linked ES Github issue, these AccessControlException warnings are not affecting the functionality of the EC2 discovery plugin. The issue must be a misconfiguration (see suggestion by @DMNSteve). In case the issue is not resolved by the above suggestion, please turn logging of "org.elasticsearch.discovery.ec2" to TRACE so we can get more insight into what's happening.

Topic		Replies	Views
Permission denied error in plugin for ES 2.2.0 Elasticsearch	6	6489	July 5, 2017
Mbeans error Elasticsearch	12	6463	August 15, 2017
JAVA Access denied - ES upgrade to 7.1 Elasticsearch	5	593	July 19, 2019
ES on AWS - com.amazonaws.AmazonClientException Elasticsearch	1	1015	July 6, 2017
Starting ElasticSearch Elasticsearch	3	1555	April 27, 2017

ES Cloud AWS error 2.2.1

Related topics