AWS Ubuntu 18, ES 6.6.0, java couldn't find discovery-ec2 plugin

Hey,

I have successfully installed ES. But JAVA is still not able to find the discovery-ec2 plugin.
Am I missing any plugin or any configuration?
below is my yml file

cloud.aws.access_key: #####
cloud.aws.secret_key: #####
discovery.type: ec2
network.host: _ec2_

cluster.name: es-clus1
node.name: searchit
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
http.port: 9200

and list of installed plugins

ubuntu@ip-172-31-19-5:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list
discovery-ec2

Here is my log

[2019-04-05T06:22:56,320][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [searchnode] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [cloud.aws.secret_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.0.jar:6.6.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [cloud.aws.secret_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:482) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:427) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:398) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:369) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:372) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
        ... 6 more
        Suppressed: java.lang.IllegalArgumentException: unknown setting [cloud.aws.access_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:482) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:427) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:398) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:369) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:148) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.node.Node.<init>(Node.java:372) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.0.jar:6.6.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.6.0.jar:6.6.0]

Here is status of elatsicsarch

ubuntu@ip-172-31-19-5:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-04-05 06:22:56 UTC; 1h 12min ago
     Docs: http://www.elastic.co
  Process: 15072 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 15072 (code=exited, status=1/FAILURE)

Apr 05 06:22:49 ip-172-31-19-5 systemd[1]: Started Elasticsearch.
Apr 05 06:22:49 ip-172-31-19-5 elasticsearch[15072]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Apr 05 06:22:56 ip-172-31-19-5 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Apr 05 06:22:56 ip-172-31-19-5 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

The error isn't that it can't find a plugin, it's this:

This is not a valid setting, although it was in the past. The client settings are rather different now.

Okk.
I have commented out it.
Thank you for replying..

But if i want use associated IP on browser and in my web application.

What configuration are required in yml?
as i tried it by appending port in browser, its shows that site is unreachable. And it show output for curl -XGET "http://127.0.0.1:9200".

I have server of AWS Ubuntu EC2.

Sorry, I'm not following. It's unlikely that you will access Elasticsearch directly from a browser, but that seems to be what you're asking about?

Yes..i am not able to access it from browser.

Is there any configuration in yml?

i have added

network.host: 0.0.0.0

Can you explain why you want to access Elasticsearch from a web browser? That's not really how it's designed to be used.

By means of using it from the browser is that,

Am I able to POST or GET requests to the elasticsearch server from my web application?

Do you mean from a Javascript application running in the browser? Yes, this is possible.

Yes..from javascript application..

I tried to send request using postman tool. But it couldn't get any response.

If i tried curl request in server and i am getting response.

It sounds like a network configuration issue, but you haven't shared much information so it's hard to give much more help. Can you give more details? The requests you're trying and the responses or errors you're seeing? More details about network connectivity between your browser and Elasticsearch? Any firewalls or other filtering?

Yes.

Server IP: ####

I am sending the request to ### by postman

Here is server response.

Firewall status of server

ubuntu@ip-172-31-19-5:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere                  
9200                       ALLOW       ###             
9200                       ALLOW       0.0.0.0                   
22 (v6)                    ALLOW       Anywhere (v6)             

elasticsearch.yml

        cluster.name: es-clus1
        node.name: searchit
        path.data: /var/lib/elasticsearch
        path.logs: /var/log/elasticsearch
        http.port: 9200
        network.host: 0.0.0.0

here I am expecting output like

ubuntu@ip-172-31-19-5:~$ curl -XGET 'http://127.0.0.1:9200'
{
  "name" : "searchit",
  "cluster_name" : "es-clus1",
  "cluster_uuid" : "D5gHXnP3Qu266GM5SsDPsA",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Thanks @kunj_choksi.

9200                       ALLOW       0.0.0.0                   

Does this really mean "allow from anywhere"? I also see Anywhere in the From column of your firewall config, so I wonder how that differs from 0.0.0.0.

It looks like port 22 is open, which might be SSH? Are you able to SSH from your local machine to this address?

Anywhere I am not 100% sure means from anywhere but i got a reference from digital ocean tutorial

And yes I am able to SSH to server from my local machine.

Hmm, that page recommends that you don't set up security, which means you'll be exposing an unsecured Elasticsearch cluster to the internet. Please don't do this. There are many bad stories of malware or data loss as a result of not properly securing your cluster.

The docs you link suggest allowing only a single IP address to contact this cluster (i.e. not 0.0.0.0), which isn't really satisfactory either, although it's ok for testing. You can find your own IP address by going to http://icanhazip.com/ , and then try using that instead of 0.0.0.0. But for production use I strongly recommend something more robust.

1 Like

Thank you for concern.
I will take care of security.
But for testing what should I do for my application?
What is blocking my request on the server?

I think it's the firewall. I think this rule does not allow access from your machine:

9200                       ALLOW       0.0.0.0                   

Instead you should set up a rule for your specific IP address.

Thank you so much.
I think the problem was with the firewall actually.
I have to disable it compulsory.

Although I tried to add the rule for my IP. But it didn't allow me to access.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.