Elasticsearch.yml chanages 2.x to 5.x on AWS EC2 - Amazon linux

murlockvoices · June 2, 2017, 9:19am

Dear Elastic,

I've decided to try out the Elasticsearch 5.x version, since I use the 2.x versions of this awesome product.
I started to install and setup on AWS EC2 Amazon linux instance, but I encountered a couple of changes in the elasticsearch.yml config file. Some of them seems undocumented or not easy to find how to implement the 2.x configs in order to work whit 5.x.

I installed the ec2-discovery plugin for elasticsearch.
Can someone help me which parts of my yml setup needs to be changed or may removed since deprecated.

Here is my 2.x yml which works like a charm:

cluster.name: ES_TestCluster
node.name: es_test_01
cloud.aws.region: eu-central-1
network.host: [ _ec2_, _local_]
plugin.mandatory: cloud-aws
Cloud.aws.access_key: ********************************
cloud.aws.secret_key: ****************************************************
discovery.type: ec2
discovery.zen.ping.multicast.enabled: false  
discovery.ec2.tag.TagGroup: ES_instance 
 
cluster.routing.allocation.awareness.attributes: zone
node.zone: zoneB
cluster.routing.allocation.awareness.force.zone.values: zoneA,zoneB
 
indices.fielddata.cache.size:  50%
gateway.recover_after_time: 5m

Thank you.

warkolm · June 2, 2017, 9:24am

What problems are you having with this?

murlockvoices · June 2, 2017, 9:39am

Thanks for your reply
I have many problems, but first of all Elasticsearch not able to start whit this setting.
Why do you asking for problems?
I assume you know very well the setup process on AWS EC2 Amazon linux and for you at a glance seems everything perfectly normal, and it should run without any problem. In this case there are a few mysteries I have to solve including that why I can't start Elasticsearch with this settings after a perfectly clean install. (Without these setting Elasticsearch runs perfectly.)

Unfortunately I'm not an expert, so any help would be awesome
Have a nice day

warkolm · June 2, 2017, 9:42am

Did you read the documentation for 5.X for the plugin to see what is and isn't valid?

That's why, and there's your mistake. I am not an expert with the EC2 discovery plugin and neither are many people here. We are happy to try to help you, but making assumptions doesn't let us do that.

murlockvoices · June 2, 2017, 9:55am

As you can see here David Pilato points out the documentation is not up to date.

Sorry for make you upset, it wasn't my intention. I hope we can still be friends

This post going to help many people in the near future cos as you said there are not many ec2-discovery expert here, and if someone going to look for a solution with a similar problem whit any keywords which are in the title, they need to see a solution or any ideas regarding this problem.

dadoonet · June 14, 2017, 8:20pm

Some comments about your settings:

Cloud.aws.access_key should be cloud.aws.access_key.
Remove discovery.zen.ping.multicast.enabled: false which is not needed anymore
node.zone should become node.attr.zone

About deprecated settings, you can keep them for now as they are or try to migrate them to secured settings.
But other things are going to change in the future so I'd probably wait for the latest elasticsearch 5.x version when 6.0 final will be released so you will be able to prepare the migration.

And I hope that everything will be documented at this point.

As I wrote in:

You will need to register the following secured settings:

discovery.ec2.access_key
discovery.ec2.secret_key

And cloud.aws.region should not be used anymore but discovery.ec2.endpoint needs to be set in elasticsearch.yml file.

I hope this helps. If you still have warning messages or errors, please share your logs so we can provide more help.

murlockvoices · June 18, 2017, 6:22pm

Hi David,

Thanks for your response and help
It was not easy but I finally managed to start Elasticsearch with Kibana on Amazon Linux.

Here is my elasticsearch.yml:

cluster.name: ES5_TestCluster
node.name: es5_test_01

network.host: [ _ec2_, _local_]

discovery.zen.hosts_provider: ec2
discovery.ec2.endpoint: eu-central-1
discovery.ec2.tag.TagGroup: ES_instance

cluster.routing.allocation.awareness.attributes: zone
node.attr.zone: zoneB
cluster.routing.allocation.awareness.force.zone.values: zoneA,zoneB

indices.fielddata.cache.size:  50%
gateway.recover_after_time: 5m

Actually this works fine however in the log I got this:

[INFO ][o.e.d.e.AwsEc2UnicastHostsProvider] [es5_test_01] Exception while retrieving instance list from AWS API: Unable to execute HTTP request: eu-central-1: Name or service not known

I was proactive again and made a little research.

The exception is generated when it calls the describeInstances() aws client function.

github.com

elastic/elasticsearch/blob/5.4/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.java#L110-L121


try {
    // Query EC2 API based on AZ, instance state, and tag.


    // NOTE: we don't filter by security group during the describe instances request for two reasons:
    // 1. differences in VPCs require different parameters during query (ID vs Name)
    // 2. We want to use two different strategies: (all security groups vs. any security groups)
    descInstances = client.describeInstances(buildDescribeInstancesRequest());
} catch (AmazonClientException e) {
    logger.info("Exception while retrieving instance list from AWS API: {}", e.getMessage());
    logger.debug("Full exception:", e);
    return discoNodes;
}

I checked that whether I'm able to run a describe function in the CLI, so I tried running the following:
aws ec2 describe-instances --region eu-central-1

It returned all the information of my instances, so it worked perfectly. It means something not quite right with my setup.

Than I saw this line in the above code:
logger.debug("Full exception:", e);

So I switched Elasticsearch to debug mode maybe I can get more information.

PUT /_cluster/settings
{
    "transient" : {
        "logger._root" : "DEBUG"
    }
}

Although the log is more detailed unfortunately I still get the same line without any more info

Additionally I get two visible exceptions in the log: (I copied just the first lines)

[DEBUG][r.suppressed ] path: /_cluster/nodes/stats, params: {pretty=, index=_cluster, id=stats, type=nodes, nodeId=stats}
java.lang.IllegalArgumentException: request [/_cluster/nodes/stats] contains unrecognized parameter: [nodeId]

[DEBUG][r.suppressed ] path: /nodes/stats, params: {pretty=, index=nodes, type=stats}
java.lang.IllegalArgumentException: No endpoint or operation is available at [stats]

If you need the full log I can upload it.

There is a part in the code where it should log some aws settings:

github.com

elastic/elasticsearch/blob/5.4/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.java#L94-L97


if (logger.isDebugEnabled()) {
    logger.debug("using host_type [{}], tags [{}], groups [{}] with any_group [{}], availability_zones [{}]", hostType, tags,
            groups, bindAnyGroup, availabilityZones);
}

BUT it's completely missing from the log.
Where can I switch on the proper debug mode in order to this function logger.isDebugEnabled() return true?

Do you have any idea regarding these issues?

Thank you for any help.

/sorry for the long post/

dadoonet · June 30, 2017, 3:06pm

The discovery.ec2.endpoint for eu-central-1 region should be something like: ec2.eu-central-1.amazonaws.com IIRC or https://ec2.eu-central-1.amazonaws.com

system · July 28, 2017, 3:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cluster discovery on Amazon EC2 problem - need urgent help Elasticsearch	11	1634	July 6, 2017
Setting ElasticSearch 5.5.0 Cluster With discovery-ec2 Plugin Elasticsearch	7	3417	August 24, 2017
Can't get ec2 discovery working Elasticsearch	4	1361	July 5, 2017
How to configure elasticsearch-2.x to make cluster in AWS machine and how is it different from older versions of ES Elasticsearch	3	452	July 5, 2017
Elasticsearch 5.3 EC2 Discovery Issue Elasticsearch	2	1374	May 27, 2017

Elasticsearch.yml chanages 2.x to 5.x on AWS EC2 - Amazon linux

Related topics