AWS Ubuntu 18, ES 6.6.0, java couldn't find discovery-ec2 plugin

kunj_choksi · April 5, 2019, 7:39am

Hey,

I have successfully installed ES. But JAVA is still not able to find the discovery-ec2 plugin.
Am I missing any plugin or any configuration?
below is my yml file

cloud.aws.access_key: #####
cloud.aws.secret_key: #####
discovery.type: ec2
network.host: _ec2_

cluster.name: es-clus1
node.name: searchit
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
http.port: 9200

and list of installed plugins

ubuntu@ip-172-31-19-5:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list
discovery-ec2

Here is my log

[2019-04-05T06:22:56,320][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [searchnode] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [cloud.aws.secret_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.0.jar:6.6.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [cloud.aws.secret_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:482) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:427) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:398) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:369) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:372) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
        ... 6 more
        Suppressed: java.lang.IllegalArgumentException: unknown setting [cloud.aws.access_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:482) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:427) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:398) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:369) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.node.Node.<init>(Node.java:372) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.6.0.jar:6.6.0]

Here is status of elatsicsarch

ubuntu@ip-172-31-19-5:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-04-05 06:22:56 UTC; 1h 12min ago
     Docs: http://www.elastic.co
  Process: 15072 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 15072 (code=exited, status=1/FAILURE)

Apr 05 06:22:49 ip-172-31-19-5 systemd[1]: Started Elasticsearch.
Apr 05 06:22:49 ip-172-31-19-5 elasticsearch[15072]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Apr 05 06:22:56 ip-172-31-19-5 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Apr 05 06:22:56 ip-172-31-19-5 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

DavidTurner · April 5, 2019, 8:15am

The error isn't that it can't find a plugin, it's this:

This is not a valid setting, although it was in the past. The client settings are rather different now.

kunj_choksi · April 5, 2019, 9:08am

Okk.
I have commented out it.
Thank you for replying..

But if i want use associated IP on browser and in my web application.

What configuration are required in yml?
as i tried it by appending port in browser, its shows that site is unreachable. And it show output for curl -XGET "http://127.0.0.1:9200".

I have server of AWS Ubuntu EC2.

DavidTurner · April 5, 2019, 9:12am

Sorry, I'm not following. It's unlikely that you will access Elasticsearch directly from a browser, but that seems to be what you're asking about?

kunj_choksi · April 5, 2019, 9:15am

Yes..i am not able to access it from browser.

Is there any configuration in yml?

i have added

network.host: 0.0.0.0

DavidTurner · April 5, 2019, 9:17am

Can you explain why you want to access Elasticsearch from a web browser? That's not really how it's designed to be used.

kunj_choksi · April 5, 2019, 9:21am

By means of using it from the browser is that,

Am I able to POST or GET requests to the elasticsearch server from my web application?

DavidTurner · April 5, 2019, 9:34am

Do you mean from a Javascript application running in the browser? Yes, this is possible.

kunj_choksi · April 5, 2019, 9:39am

Yes..from javascript application..

I tried to send request using postman tool. But it couldn't get any response.

If i tried curl request in server and i am getting response.

DavidTurner · April 5, 2019, 9:41am

It sounds like a network configuration issue, but you haven't shared much information so it's hard to give much more help. Can you give more details? The requests you're trying and the responses or errors you're seeing? More details about network connectivity between your browser and Elasticsearch? Any firewalls or other filtering?

kunj_choksi · April 5, 2019, 9:59am

Yes.

Server IP: ####

I am sending the request to ### by postman

Here is server response.

Firewall status of server

ubuntu@ip-172-31-19-5:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere                  
9200                       ALLOW       ###             
9200                       ALLOW       0.0.0.0                   
22 (v6)                    ALLOW       Anywhere (v6)

elasticsearch.yml

        cluster.name: es-clus1
        node.name: searchit
        path.data: /var/lib/elasticsearch
        path.logs: /var/log/elasticsearch
        http.port: 9200
        network.host: 0.0.0.0

here I am expecting output like

ubuntu@ip-172-31-19-5:~$ curl -XGET 'http://127.0.0.1:9200'
{
  "name" : "searchit",
  "cluster_name" : "es-clus1",
  "cluster_uuid" : "D5gHXnP3Qu266GM5SsDPsA",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

DavidTurner · April 5, 2019, 10:15am

Thanks @kunj_choksi.

9200                       ALLOW       0.0.0.0

Does this really mean "allow from anywhere"? I also see Anywhere in the From column of your firewall config, so I wonder how that differs from 0.0.0.0.

It looks like port 22 is open, which might be SSH? Are you able to SSH from your local machine to this address?

kunj_choksi · April 5, 2019, 10:21am

Anywhere I am not 100% sure means from anywhere but i got a reference from digital ocean tutorial

And yes I am able to SSH to server from my local machine.

DavidTurner · April 5, 2019, 10:35am

Hmm, that page recommends that you don't set up security, which means you'll be exposing an unsecured Elasticsearch cluster to the internet. Please don't do this. There are many bad stories of malware or data loss as a result of not properly securing your cluster.

The docs you link suggest allowing only a single IP address to contact this cluster (i.e. not 0.0.0.0), which isn't really satisfactory either, although it's ok for testing. You can find your own IP address by going to http://icanhazip.com/ , and then try using that instead of 0.0.0.0. But for production use I strongly recommend something more robust.

kunj_choksi · April 5, 2019, 11:09am

Thank you for concern.
I will take care of security.
But for testing what should I do for my application?
What is blocking my request on the server?

DavidTurner · April 5, 2019, 11:10am

I think it's the firewall. I think this rule does not allow access from your machine:

9200                       ALLOW       0.0.0.0

Instead you should set up a rule for your specific IP address.

kunj_choksi · April 5, 2019, 12:15pm

Thank you so much.
I think the problem was with the firewall actually.
I have to disable it compulsory.

Although I tried to add the rule for my IP. But it didn't allow me to access.

system · May 3, 2019, 12:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ClassNotFoundException: org.elasticsearch.discovery.ec2.Ec2DiscoveryModule Elasticsearch	3	652	July 6, 2017
Unable to install discovery-ec2 plugin Elasticsearch	4	1186	July 5, 2017
Cloud-aws for ES 2.1.1 in maven Elasticsearch	12	2560	July 5, 2017
Cloud-aws not working 0.20.1 and 0.19.12? Elasticsearch	5	408	July 6, 2017
EC2 Discovery Pluginにおけるcloud.aws.regionの変更について日本語による質問・議論はこちら	5	1452	July 11, 2017

AWS Ubuntu 18, ES 6.6.0, java couldn't find discovery-ec2 plugin

Related topics