Elasticsearch.yml chanages 2.x to 5.x on AWS EC2 - Amazon linux

Dear Elastic,

I've decided to try out the Elasticsearch 5.x version, since I use the 2.x versions of this awesome product.
I started to install and setup on AWS EC2 Amazon linux instance, but I encountered a couple of changes in the elasticsearch.yml config file. Some of them seems undocumented or not easy to find how to implement the 2.x configs in order to work whit 5.x.

I installed the ec2-discovery plugin for elasticsearch.
Can someone help me which parts of my yml setup needs to be changed or may removed since deprecated.

Here is my 2.x yml which works like a charm:

cluster.name: ES_TestCluster
node.name: es_test_01
cloud.aws.region: eu-central-1
network.host: [ _ec2_, _local_]
plugin.mandatory: cloud-aws
Cloud.aws.access_key: ********************************
cloud.aws.secret_key: ****************************************************
discovery.type: ec2
discovery.zen.ping.multicast.enabled: false  
discovery.ec2.tag.TagGroup: ES_instance 
 
cluster.routing.allocation.awareness.attributes: zone
node.zone: zoneB
cluster.routing.allocation.awareness.force.zone.values: zoneA,zoneB
 
indices.fielddata.cache.size:  50%
gateway.recover_after_time: 5m

Thank you. :slight_smile:

What problems are you having with this?

Thanks for your reply :slight_smile:
I have many problems, but first of all Elasticsearch not able to start whit this setting.
Why do you asking for problems?
I assume you know very well the setup process on AWS EC2 Amazon linux and for you at a glance seems everything perfectly normal, and it should run without any problem. In this case there are a few mysteries I have to solve including that why I can't start Elasticsearch with this settings after a perfectly clean install. (Without these setting Elasticsearch runs perfectly.)

Unfortunately I'm not an expert, so any help would be awesome :slight_smile:
Have a nice day :slight_smile:

Did you read the documentation for 5.X for the plugin to see what is and isn't valid?

That's why, and there's your mistake. I am not an expert with the EC2 discovery plugin and neither are many people here. We are happy to try to help you, but making assumptions doesn't let us do that.

As you can see here David Pilato points out the documentation is not up to date.

Sorry for make you upset, it wasn't my intention. I hope we can still be friends :slight_smile:

This post going to help many people in the near future cos as you said there are not many ec2-discovery expert here, and if someone going to look for a solution with a similar problem whit any keywords which are in the title, they need to see a solution or any ideas regarding this problem.

Some comments about your settings:

  • Cloud.aws.access_key should be cloud.aws.access_key.
  • Remove discovery.zen.ping.multicast.enabled: false which is not needed anymore
  • node.zone should become node.attr.zone

About deprecated settings, you can keep them for now as they are or try to migrate them to secured settings.
But other things are going to change in the future so I'd probably wait for the latest elasticsearch 5.x version when 6.0 final will be released so you will be able to prepare the migration.

And I hope that everything will be documented at this point.

As I wrote in:

You will need to register the following secured settings:

  • discovery.ec2.access_key
  • discovery.ec2.secret_key

And cloud.aws.region should not be used anymore but discovery.ec2.endpoint needs to be set in elasticsearch.yml file.

I hope this helps. If you still have warning messages or errors, please share your logs so we can provide more help.

1 Like

Hi David,

Thanks for your response and help :slight_smile:
It was not easy but I finally managed to start Elasticsearch with Kibana on Amazon Linux.

Here is my elasticsearch.yml:

cluster.name: ES5_TestCluster
node.name: es5_test_01

network.host: [ _ec2_, _local_]

discovery.zen.hosts_provider: ec2
discovery.ec2.endpoint: eu-central-1
discovery.ec2.tag.TagGroup: ES_instance

cluster.routing.allocation.awareness.attributes: zone
node.attr.zone: zoneB
cluster.routing.allocation.awareness.force.zone.values: zoneA,zoneB

indices.fielddata.cache.size:  50%
gateway.recover_after_time: 5m

Actually this works fine however in the log I got this:

[INFO ][o.e.d.e.AwsEc2UnicastHostsProvider] [es5_test_01] Exception while retrieving instance list from AWS API: Unable to execute HTTP request: eu-central-1: Name or service not known

I was proactive again and made a little research.

The exception is generated when it calls the describeInstances() aws client function.

I checked that whether I'm able to run a describe function in the CLI, so I tried running the following:
aws ec2 describe-instances --region eu-central-1

It returned all the information of my instances, so it worked perfectly. It means something not quite right with my setup.

Than I saw this line in the above code:
logger.debug("Full exception:", e);

So I switched Elasticsearch to debug mode maybe I can get more information.

PUT /_cluster/settings
{
    "transient" : {
        "logger._root" : "DEBUG"
    }
}

Although the log is more detailed unfortunately I still get the same line without any more info :confused:

Additionally I get two visible exceptions in the log: (I copied just the first lines)

[DEBUG][r.suppressed ] path: /_cluster/nodes/stats, params: {pretty=, index=_cluster, id=stats, type=nodes, nodeId=stats}
java.lang.IllegalArgumentException: request [/_cluster/nodes/stats] contains unrecognized parameter: [nodeId]

[DEBUG][r.suppressed ] path: /nodes/stats, params: {pretty=, index=nodes, type=stats}
java.lang.IllegalArgumentException: No endpoint or operation is available at [stats]

If you need the full log I can upload it.

There is a part in the code where it should log some aws settings:

BUT it's completely missing from the log. :confused:
Where can I switch on the proper debug mode in order to this function logger.isDebugEnabled() return true?

Do you have any idea regarding these issues?

Thank you for any help. :slight_smile:

/sorry for the long post/

The discovery.ec2.endpoint for eu-central-1 region should be something like: ec2.eu-central-1.amazonaws.com IIRC or https://ec2.eu-central-1.amazonaws.com

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.