Ok thanks David. Let me know if you have any conference scheduled in near future in UK. I would like to attend the same.
Regarding the queries :- My cluster is still not allowing me to write any new indices as it is blocked by Cluster Write Blocked. The only reason I see is the JVM memory pressure is too high around 95%.
Things I am struggling to figure out :
My Cluster has 2 nodes each with 512GB disk space and 8 GB RAM.
My default settings are 5 Primary shards and 1 replica shard.
When I query for a particular index GET _cat/shards/index_name it gives me 10 records with the allocation like 2 primary 2 replica for certain no. of documents and 4p,4r for another set of documents and 3p,3r for another set , 1p,1r for another sets and 0p,0r for another sets.
Why the allocation is like this as described below :
Index_name 2 r STARTED 925 612.3kb x.x.x.x replica node name
Index_name 2 p STARTED 925 612.3kb x.x.x.x primary node name
Index_name 4 r STARTED 910 568.5kb x.x.x.x replica node name
Index_name 4 p STARTED 910 568.5kb x.x.x.x primary node name
Index_name 3 r STARTED 907 615.1kb x.x.x.x replica node name
Index_name 3 p STARTED 907 615.1kb x.x.x.x primary node name
Index_name 1 r STARTED 881 481.7kb x.x.x.x replica node name
Index_name 1 p STARTED 881 481.7kb x.x.x.x primary node name
Index_name 0 r STARTED 920 631.2kb x.x.x.x replica node name
Index_name 0 p STARTED 920 631.2kb x.x.x.x primary node name
I have read somewhere that fielddata put lot of pressure of JVM heap so tried to clear the cache of all fielddata that did not work out.Also the no.of fielddata is not that huge. I have updated the fielddata cahce size to 40%
I am actually stuck now as I am not able to decide what configuration do I need to update so that writing new indexes is enabled for now and also I am looking for a long term solution to update the cluster in production .
Do I need to change the primary shard value ?
And how do you determine the heap size looking at the above index size ?
Any quick check to determine what is participating my JVM memory pressure for which the write operation got blocked ?
Please let me know the settings to get rid of this problem now.