ELK 6.3.1 - how to delete _cluster/settings

Elastic Stack Elasticsearch
1

Hello,

I am facing issue with updating cluster settings in ES 6.3.1

I tried below command

PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "100gb",
"cluster.routing.allocation.disk.watermark.high": "50gb",
"cluster.info.update.interval": "1m"
}
}

Output

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[elk-mgm-node3][10.138.0.8:9300][cluster:admin/settings/update]"
}
],
"type": "illegal_argument_exception",
"reason": "unable to consistently parse [cluster.routing.allocation.disk.watermark.low=100gb], [cluster.routing.allocation.disk.watermark.high=50gb], and [cluster.routing.allocation.disk.watermark.flood_stage=95%] as percentage or bytes",
"caused_by": {
"type": "parse_exception",
"reason": "failed to parse setting [cluster.routing.allocation.disk.watermark.flood_stage] with value [95%] as a size in bytes: unit is missing or unrecognized"
}
},
"status": 400
}

Current cluster settings are as below

{
"persistent": {
"archived": {
"indices": {
"store": {
"throttle": {
"max_bytes_per_sec": "10gb"
}
}
}
},
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": "8"
}
}
}
},
"transient": {
"archived": {
"cluster": {
"routing": {
"allocation": {
"disk": {
"watermark": {
"low": "100gb",
"high": "50gb"
}
}
}
}
}
},
"cluster": {
"routing": {
"allocation": {
"disk": {
"watermark": {
"low": "100gb",
"flood_stage": "10gb",
"high": "50gb"
}
},
"enable": "all"
}
},
"info": {
"update": {
"interval": "1m"
}
}
}
}
}

Is there any way to delete these settings

  1. Related issue

When I try to turn on monitoring in ELK 6.3.1 I get below error on UI

400 Bad Request
[remote_transport_exception] [elk-mgm-node3][10.138.0.8:9300][cluster:admin/settings/update]

(Recently upgraded ELK from 5.6 to 6.3)

2

what is yous elasticsearch.yml ?

3

elasticsearch.yml from management node ?

4

Yes.

5

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please see the documentation for further information on configuration options:

http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: cluster_name

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: elk-mgm-node

Add custom attributes to the node:

node.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: path

Path to log files:

path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.mlockall: true
bootstrap.memory_lock: true
bootstrap.system_call_filter: false

Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory

available on the system and that the owner of the process is allowed to use this limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.bind_host: 0.0.0.0

Set a custom port for HTTP:

#net.publish_host: ip
network.host: ip
http.port: 9200

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["host1", "host2"]

Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 3

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

gateway.recover_after_nodes: 3

For more information, see the documentation at:

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html

---------------------------------- Various -----------------------------------

Disable starting multiple nodes on a single system:

node.max_local_storage_nodes: 1

Require explicit names when deleting indices:

action.destructive_requires_name: true

discovery.zen.minimum_master_nodes: 2
#discovery.zen.ping.timeout: 120s
discovery.zen.ping_timeout: 120s
#discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["ip_list"]
node.master: true
node.data: false
#gateway.recover_after_nodes: 8
#gateway.expected_nodes: 10
#gateway.recover_after_time: 5m

6

@pastechecker
is there any direct way to force cluster settings ?
Due to this I am not able to change routing as well

PUT /_cluster/settings
{ "transient": { "cluster.routing.allocation.enable": "all" }}

Output

{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[elk-mgm-node3][10.138.0.8:9300][cluster:admin/settings/update]"
}
],
"type": "illegal_argument_exception",
"reason": "unknown setting [archived.cluster.routing.allocation.disk.watermark.high] did you mean any of [cluster.routing.allocation.disk.watermark.high, cluster.routing.allocation.disk.watermark.low]?",
"suppressed": [
{
"type": "illegal_argument_exception",
"reason": "unknown setting [archived.cluster.routing.allocation.disk.watermark.low] did you mean any of [cluster.routing.allocation.disk.watermark.low, cluster.routing.allocation.disk.watermark.high]?"
}
]
},
"status": 400
}

7

I cannot help you, that is beyond my current understanding and my experience.
You need to go deeper into upgrade process between versions.

8

Hey, No problem. Thanks a lot.

I know one way is to delete global_state file and take full cluster restart on each node
But I am searching for "api" way to change these settings

@elastic Hello, it would be a great help if I get some clue on this !

9

I solved it,
put to null state all the settings that are showing in archived.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.