# Node cluster

**URL:** <https://discuss.elastic.co/t/node-cluster/143650>\
**Category:** Elasticsearch\
**Created:** [August 9, 2018, 8:37am UTC](https://discuss.elastic.co/t/node-cluster/143650 "2018-08-09T08:37:22Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 8:37am UTC](https://discuss.elastic.co/t/node-cluster/143650/1 "2018-08-09T08:37:22Z")

</div>

Hello everyone,

What could happen if in my cluster, all nodes are data nodes?  
Do we have to have, within the cluster, a master node to manage the other nodes?

```
"nodes" : {
    "count" : {
      "total" : 4,
      "master_only" : 0,
      "data_only" : 0,
      "master_data" : 4,
      "client" : 0
    },

```

Thank you.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 9, 2018, 9:00am UTC](https://discuss.elastic.co/t/node-cluster/143650/2 "2018-08-09T09:00:01Z")

</div>

For a 4 nodes cluster, everything is correct here.  
You have 4 nodes that can store/index data. You have 4 nodes that can become master if the current master node is failing.

Just don't forget to define the `discovery.zen.minimum_master_nodes` to `3` in your `elasticsearch.yml` files.

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:03am UTC](https://discuss.elastic.co/t/node-cluster/143650/3 "2018-08-09T09:03:13Z")

</div>

No, I did not forget it. By cons, I see a small failure in my dashboards. Too slow and sometimes it fails.

RAM is 80% used

What should I change?

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:07am UTC](https://discuss.elastic.co/t/node-cluster/143650/4 "2018-08-09T09:07:02Z")

</div>

```
"process" : {
      "cpu" : {
        "percent" : 70
      },
      "open_file_descriptors" : {
        "min" : 1700,
        "max" : 1748,
        "avg" : 1729
      }
    },
  
      "mem" : {
        "heap_used" : "36.6gb",
        "heap_used_in_bytes" : 39362479080,
        "heap_max" : "47.8gb",
        "heap_max_in_bytes" : 51400146944
      },
      "threads" : 280
    },
    "fs" : {
      "total" : "2.5tb",
      "total_in_bytes" : 2834552881152,
      "free" : "1.1tb",
      "free_in_bytes" : 1242280148992,
      "available" : "1022.8gb",
      "available_in_bytes" : 1098293264384,
      "spins" : "true"
    },
```

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Merlo](https://avatars.discourse-cdn.com/v4/letter/g/a88e57/32.png) [@Giuseppe\_Merlo](https://discuss.elastic.co/u/Giuseppe_Merlo)\
**Post date:** [August 9, 2018, 9:24am UTC](https://discuss.elastic.co/t/node-cluster/143650/5 "2018-08-09T09:24:48Z")

</div>

Hi!

Can you tell me what api return this?

> [@Charaf\_Ahmed](#):
>
> "nodes" : { "count" : { "total" : 4, "master\_only" : 0, "data\_only" : 0, "master\_data" : 4, "client" : 0 },

I have started 2 nodes in my elasticsearch cluster, on the same machine both, but it seems there is an error on shards allocation.

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:27am UTC](https://discuss.elastic.co/t/node-cluster/143650/6 "2018-08-09T09:27:21Z")

</div>

> [@Giuseppe\_Merlo](#):
>
> Can you tell me what api return this?

`

`curl -X GET "IP:9200/_cluster/stats?human&pretty"`

`

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Merlo](https://avatars.discourse-cdn.com/v4/letter/g/a88e57/32.png) [@Giuseppe\_Merlo](https://discuss.elastic.co/u/Giuseppe_Merlo)\
**Post date:** [August 9, 2018, 9:29am UTC](https://discuss.elastic.co/t/node-cluster/143650/7 "2018-08-09T09:29:01Z")

</div>

For this question, have you tried to use this?

```
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%"
  }
}

```

i don't think it resolves your problem but you can try

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:32am UTC](https://discuss.elastic.co/t/node-cluster/143650/8 "2018-08-09T09:32:41Z")

</div>

```
 curl -X GET "IP:9200/_cluster/settings?human&pretty"
{
  "persistent" : { },
  "transient" : { }
}
```

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Merlo](https://avatars.discourse-cdn.com/v4/letter/g/a88e57/32.png) [@Giuseppe\_Merlo](https://discuss.elastic.co/u/Giuseppe_Merlo)\
**Post date:** [August 9, 2018, 9:37am UTC](https://discuss.elastic.co/t/node-cluster/143650/9 "2018-08-09T09:37:30Z")

</div>

It seems you don't have any persistent or transient settings? Is it?

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:39am UTC](https://discuss.elastic.co/t/node-cluster/143650/10 "2018-08-09T09:39:54Z")

</div>

Yes exactly. But this has no direct link with the problem I currently have

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Merlo](https://avatars.discourse-cdn.com/v4/letter/g/a88e57/32.png) [@Giuseppe\_Merlo](https://discuss.elastic.co/u/Giuseppe_Merlo)\
**Post date:** [August 9, 2018, 9:41am UTC](https://discuss.elastic.co/t/node-cluster/143650/11 "2018-08-09T09:41:44Z")

</div>

Yes, but i don't know why your dashboard fails.

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 9:55am UTC](https://discuss.elastic.co/t/node-cluster/143650/12 "2018-08-09T09:55:29Z")

</div>

I suppose that the problem could come from the fact that all my nodes are of type data. I do not really know.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 9, 2018, 10:01am UTC](https://discuss.elastic.co/t/node-cluster/143650/13 "2018-08-09T10:01:19Z")

</div>

> No, I did not forget it. By cons, I see a small failure in my dashboards. Too slow and sometimes it fails.

Can you share Elasticsearch logs?

What is the output of:

```auto
GET /_cat/nodes?v
GET /_cat/indices?v
GET /_cat/health?v

```

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 10:06am UTC](https://discuss.elastic.co/t/node-cluster/143650/14 "2018-08-09T10:06:23Z")

</div>

```
curl -X GET "XXXXXXXX1:9200/_cat/health?v"
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1533809057 12:04:17 PR1_elasticsearch green 4 4 273 273 0 0 0 0 - 100.0%

curl -X GET "xxxxxxxx:9200/_cat/nodes?v"
host ip heap.percent ram.percent load node.role master name
xxxxxxx xxxxxxxx 75 99 1.03 d m xxxxx
xxxxxxx xxxxxxxx 76 99 1.15 d * xxxxxx
xxxxxxx xxxxxxxx 77 96 0.96 d m xxxxxx
xxxxxxx xxxxxxxx 77 96 0.83 d m xxxxxxx
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 9, 2018, 11:08am UTC](https://discuss.elastic.co/t/node-cluster/143650/15 "2018-08-09T11:08:38Z")

</div>

When you replace values, please make sure to replace by the same number of characters to preserve indentation. This is hard to read here.

You did not share:

```auto
GET /_cat/indices?v

```

Please do.

Also:

```auto
GET /_cat/nodes?v&h=id,hc,hm,rc,rm,r

```

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 11:43am UTC](https://discuss.elastic.co/t/node-cluster/143650/16 "2018-08-09T11:43:54Z")

</div>

```
 curl -X GET "IP1:9200/_cat/indices?v"
health status index pri rep docs.count docs.deleted store.size pri.store.size
       close idx-pr1-2018.05.31
green open idx-pr1-2018.07.10 4 0 26810265 233580 27.2gb 27.2gb
green open idx-pr1-2018.07.12 4 0 44746353 0 41.4gb 41.4gb
green open idx-pr1-2018.07.11 4 0 24837757 0 24.3gb 24.3gb
green open idx-pr1-2018.07.14 4 0 25572880 0 22gb 22gb
green open idx-pr1-2018.07.13 4 0 38307116 0 37.6gb 37.6gb
       close idx-pr1-2018.05.30
green open flex2gateway 4 0 0 0 636b 636b
green open idx-pr1-2018.07.15 4 0 23033981 0 19.8gb 19.8gb
green open idx-pr1-2018.07.16 4 0 27849293 0 26.4gb 26.4gb
green open idx-pr1-2018.07.17 4 0 29240119 0 29gb 29gb
green open idx-pr1-2018.07.18 4 0 31652960 0 32.2gb 32.2gb
green open idx-pr1-2018.07.19 4 0 34845354 0 33.5gb 33.5gb
green open sawmill6cl.exe 4 0 0 0 636b 636b
green open idx-pr1-2018.07.21 4 0 29075955 0 24.1gb 24.1gb
green open idx-pr1-2018.07.20 4 0 31747500 0 29.7gb 29.7gb
green open idx-pr1-2018.07.25 4 0 28928470 0 27.8gb 27.8gb
green open idx-pr1-grokparsefailure-2018.07.31 4 0 13483 0 4.2mb 4.2mb
 .........
green open idx-pr1-2018.08.07 4 0 18751654 0 17.5gb 17.5gb
green open idx-pr1-2018.08.08 4 0 28011365 0 25.7gb 25.7gb
green open idx-pr1-2018.08.05 4 0 30826869 0 24.9gb 24.9gb
green open idx-pr1-2018.08.06 4 0 22930264 0 21.6gb 21.6gb
green open webui 4 0 0 0 636b 636b
green open idx-pr1-2018.07.30 4 0 51936723 0 43.7gb 43.7gb
green open idx-pr1-2018.07.31 4 0 56453887 0 47.9gb 47.9gb
green open idx-pr1-2018.08.09 4 0 15500095 0 17.5gb 17.5gb
green open .kibanapr1 1 0 3020 7 1.4mb 1.4mb
green open perl 4 0 0 0 636b 636b
green open idx-pr1-2018.08.04 4 0 51318754 0 42.5gb 42.5gb
green open idx-pr1-2018.08.03 4 0 39364728 0 37.3gb 37.3gb
green open idx-pr1-2018.08.02 4 0 39895818 0 37gb 37gb
green open idx-pr1-2018.08.01 4 0 47127645 0 41.8gb 41.8gb
       close idx-pr1-2018.05.14
green open idx-pr1-grokparsefailure-2018.08.05 4 0 76514 0 13mb 13mb
green open idx-pr1-grokparsefailure-2018.08.04 4 0 78134 0 13.8mb 13.8mb
       close idx-pr1-2018.05.13
green open idx-pr1-grokparsefailure-2018.08.03 4 0 77126 0 14mb 14mb
       close idx-pr1-2018.05.16
green open idx-pr1-grokparsefailure-2018.08.02 4 0 24745 0 5.5mb 5.5mb
       close idx-pr1-2018.05.15
       close idx-pr1-2018.05.18
green open idx-pr1-grokparsefailure-2018.08.09 4 0 36836 0 8.6mb 8.6mb
       close idx-pr1-2018.05.17
green open idx-pr1-grokparsefailure-2018.08.08 4 0 78418 0 14.6mb 14.6mb
green open idx-pr1-grokparsefailure-2018.08.07 4 0 77155 0 14.5mb 14.5mb
       close idx-pr1-2018.05.19
green open idx-pr1-grokparsefailure-2018.08.06 4 0 79260 0 15.2mb 15.2mb
green open idx-pr1-2018.07.03 4 0 34240138 0 32.8gb 32.8gb
green open idx-pr1-2018.07.02 4 0 43707829 0 38.9gb 38.9gb
green open idx-pr1-2018.07.01 4 0 8913476 0 7.3gb 7.3gb
       close idx-pr1-2018.05.22
       close idx-pr1-2018.05.23
       close idx-pr1-2018.05.20
green open idx-pr1-grokparsefailure-2018.08.01 4 0 1537 0 4.7mb 4.7mb
       close idx-pr1-2018.05.21
green open lcds 4 0 0 0 636b 636b
green open blazeds 4 0 0 0 636b 636b
       close idx-pr1-2018.05.27
       close idx-pr1-2018.05.26
       close idx-pr1-2018.05.25
green open idx-pr1-2018.07.08 4 0 17424378 0 14.3gb 14.3gb
green open idx-pr1-2018.07.09 4 0 26994642 0 25.5gb 25.5gb
       close idx-pr1-2018.05.24
green open idx-pr1-2018.07.06 4 0 36173805 0 33.7gb 33.7gb
green open idx-pr1-2018.07.07 4 0 23657449 0 21.8gb 21.8gb
green open idx-pr1-2018.07.04 4 0 25739564 0 25.2gb 25.2gb
       close idx-pr1-2018.05.29
green open messagebroker 4 0 0 0 636b 636b
       close idx-pr1-2018.05.28
green open idx-pr1-2018.07.05 4 0 25102937 0 24.6gb 24.6gb
```

---

<div class="post-metadata">

**Author:** ![Charaf\_Ahmed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charaf_ahmed/32/30467_2.png) [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\
**Post date:** [August 9, 2018, 11:44am UTC](https://discuss.elastic.co/t/node-cluster/143650/17 "2018-08-09T11:44:12Z")

</div>

> [@dadoonet](#):
>
> ```auto
> GET /_cat/nodes?v&h=id,hc,hm,rc,rm,r
> 
> ```

```
curl -X GET "IP:9200/_cat/nodes?v&h=id,hc,hm,rc,rm,r"
id hc hm rc rm r
nceF 9gb 11.9gb 14.6gb 15.5gb d
JQgq 9.2gb 11.9gb 15.1gb 15.5gb d
5_Em 9.2gb 11.9gb 15gb 15.5gb d
gNzA 9.1gb 11.9gb 14.9gb 15.5gb d

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 9, 2018, 1:54pm UTC](https://discuss.elastic.co/t/node-cluster/143650/18 "2018-08-09T13:54:25Z")

</div>

You might probably have too many shards per node.

May I suggest you look at the following resources about sizing:

[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

> **[How many shards should I have in my Elasticsearch cluster?
	  	 | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...

> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...

In most if not all cases, your daily indices could probably be ok with one single or 2 shards instead of 4.

You also have a lot of closed index which is ok but may be you'd like to remove old indices?  
You set the HEAP to 12gb on a 16gb machine. You should not use more than the half of available RAM. So heap should be 8gb at most.  
May be you could start new nodes if you still need to keep all the data you have though?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 6, 2018, 1:54pm UTC](https://discuss.elastic.co/t/node-cluster/143650/19 "2018-09-06T13:54:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.