I am setting up a cluster on ES version 6.2.3 and i have following scenario:
- Data Volume 80GB per day (with 1 month of retention )
- Search scenarios (dashboards only few aggregation queries, max 50 users a day)
Current Configuration -
- 3 master eligible nodes (8 GB RAM , 2 Core CPU, 100 GB Hard Disk each each)
- 10 Data Nodes (8 GB RAM , 2 Core CPU, 512 GB (total to retain data for 1 month) )
There are two possible scenarios
- Data nodes also act as node with HTTP enable or
- Take separate client nodes with HTTP enable, and NODE_DATA as false
Other settings -
- The Java process XMX is set to 3500m (as 50% was recommended).
- Shards per indexes = 5 (default)
- Replication per shard = 2
- Master nodes are NOT data node
- refresh interval for indices set to be 30s
- For each day, data will be stored in separate index.
Is this config set up good for my scenario ?
Problems we faced while testing -
- Encountered Http code 502 from http APIs while load testing with this config
- Sometimes nodes are going down
- Few of the shards coming as UNASSIGNED
What can be the reasons for these issues? What should we monitor or change to do in config?