There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
What is the output of:
If some outputs are too big, please share them on gist.github.com and link them here.
I feel the problem is in elasticsearch.yml configuration , its because if I am stopping elasticsearch service in any of the node or bringing down any of the node still my cluster status is showing "green" for example please find the below link , I hope you got my point now what I am trying to explain kindly share your opinion
I think you misunderstand what it means for an Elasticsearch cluster to be in a "green" state.
For an Elasticsearch cluster to be green all its shards, that is all the data in all the indices you've created plus all the replica shards too, must be available for queries (and for indexing). It doesn't matter how many nodes you have, as long as you have a sufficient number to spread the primary and replica shards (you can't have a replica on the same node where the primary shard is, so you need at least 2 data nodes for a well formed, redundant cluster).
So, when you stop a node your cluster will still turn green because the shards on the stopped node will get replicated to the other nodes in the cluster. This replication may take time, during which the cluster will be in a yellow state, but once all shards are successfully replicated your cluster will be green by definition. The missing nodes have no effect because all the data is available.
The Elasticsearch master node have no memories of former nodes, only of the nodes currently in the cluster, so if you stop one or more nodes the master will simply remove them from its cluster state and won't come looking for them again. From the master's perspective the cluster is OK (green) as soon as all shards from the stopped nodes have been replicated. And that's the only sane definition of green / OK.
That is correct. If all the shards , primary and replica, can be distributed on the remaining data nodes the cluster will be in a green state once they are all synched and available.
As far as i can see there is no mention of nodes with respect to cluster status in the document you refer to, what that document says about cluster level status is:
The cluster status is controlled by the worst index status.
So the cluster's health is determined by the worst index which again is determined by the worst shard. So if the worst index is missing a replica it is yellow, which means the cluster is in yellow. But if the worst index is missing a primary, which can happen if you stop two data nodes with one of the nodes holding the primary and the other node the replica shard, then that index is in a red state which means the cluster is also in a red state.
Notice that the number of nodes has nothing to do with how the state of the cluster is computed, only the shards. So as long as you keep the shards well distributed your cluster will stay green even if you have stopped one or more nodes
It's hard to give a good answer as I don't know your use cases, your hardware and CPU specs or even which Elasticsearch version you're using. But in general:
3 master eligible nodes are sufficient, as that is the minimum to find quorum during master election. So i would change one of the master eligible nodes to be a data node (node.data; true) and set discovery.zen.minimum_master_nodes: 3 in the Elasticsearch.yml files.
I haven't set bootstrap.memory_lock: true in any of my clusters so I have no experience with this setting, but be aware of this warning in the official docs that mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available.
The high and low disk watermarks depends on how much disk space you have on your data nodes and on your use cases. For me it's more natural to think percentage than GB, but that's just a matter of preference as I always try to keep the disk usage below 75% on my data nodes, in case I need some extra space for re-indexing or if one of the data nodes falls out of the cluster and its shards get relocated to the other data nodes.
By the way, I think you've got the low, high and flood_stage watermarks backwards - the values should be low < high < flood_stage since the low watermark marks when the data node won't accept more shards, the high watermark marks when the data node will start to move away shards to free space and the flood_stage watermark marks a critically full disk (95% by default) when Elasticsearch will mark indices as read-only to stop more data from being indexed on that node.
For me the default watermark levels, 85% for low, 90% for high and 95% for flood_stage, are usually good enough so I rarely change these in my Elasticsearch.yml files.
I hope this helps you configure your cluster. Good luck!