ElasticSearch Setup on EC2

Elastic Stack Elasticsearch
1

I am trying to setup ES cluster( 3 node) on EC2. I have tried lot of configs but ended with this config:

> cluster.name: ES-CLUSTER

> node.name: ES-NODE-01

> node.max_local_storage_nodes: 1

> node.rack_id: rack_us_east_1d

> index.number_of_shards: 16

> index.number_of_replicas: 1

> gateway.recover_after_nodes: 3

> gateway.recover_after_time: 2m

> gateway.expected_nodes: 3

> discovery.zen.minimum_master_nodes: 3

> discovery.zen.ping.multicast.enabled: false

> cloud.aws.access_key: ********
> cloud.aws.secret_key: ******

> cloud.aws.region: us-east-1

> discovery.type: ec2

> discovery.ec2.groups: groups-name

> plugin.mandatory: cloud-aws,analysis-icu,analysis-phonetic


> http.cors.enabled: true

> http.cors.allow-origin: "*"


> network.host: _ec2:privateIpv4_

> network.bind_host: 0.0.0.0

> network.publish_host: _ec2:privateIpv4_

Now it is working. I am not sure it is right setting or not ?

Can anybody guide me what is right way t do this ?
Thanks

2

That's probably too many.

3

And also be aware that you won't be able to define index.number_of_shards in elasticsearch.yml in 5.0. You should use an index template for that.

1 Like
4

While we are at it, it's Elasticsearch, not ElasticSearch. That's definitely a best practise :wink:

5

What are the recommended number for number_of_shards ?
Our use case are 20 parallel requests where each request average 10k documents.
@warkolm Ok, Next time I would follow it.

6

What sort of requests would need that many docs in one?

7

one request contains one CSV data.
@warkolm

8

You mean you are bulk indexing these?

9

yes.