ES seems to be stuck in resyncing mappings

Anurag_Phadke · July 26, 2011, 1:41am

We added 3 more nodes to our cluster and increased the replication
from 0 to 1. The following WARN msgs are coming in the log files:
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110703]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110520]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110704]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110608]
re-syncing mappings with cluster state for types [[crash_reports]]

This has been going for almost an hour now:
curl -XGET http://localhost:9200/_cluster/health
{"cluster_name":"socorro","status":"red","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":7,"active_primary_shards":457,"active_shards":457,"relocating_shards":1,"initializing_shards":0,"unassigned_shards":8}

returns red.

curl -XGET 'http://localhost:9200/_cluster/state?pretty=true&filter_nodes=true&filter_indices=true'
returns everything in:
"state" : "STARTED",

any idea what might be going wrong here?

-anurag

kimchy · July 26, 2011, 4:58am

Are you using the same version across the cluster?

On Tue, Jul 26, 2011 at 4:41 AM, Anurag anurag.phadke@gmail.com wrote:

We added 3 more nodes to our cluster and increased the replication
from 0 to 1. The following WARN msgs are coming in the log files:
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110703]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110520]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110704]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110608]
re-syncing mappings with cluster state for types [[crash_reports]]

This has been going for almost an hour now:
curl -XGET http://localhost:9200/_cluster/health

{"cluster_name":"socorro","status":"red","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":7,"active_primary_shards":457,"active_shards":457,"relocating_shards":1,"initializing_shards":0,"unassigned_shards":8}

returns red.

curl -XGET '
http://localhost:9200/_cluster/state?pretty=true&filter_nodes=true&filter_indices=true
'
returns everything in:
"state" : "STARTED",

any idea what might be going wrong here?

-anurag

Anurag_Phadke · July 26, 2011, 1:46pm

yes - 0.17.1

On Mon, Jul 25, 2011 at 9:58 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

Are you using the same version across the cluster?

On Tue, Jul 26, 2011 at 4:41 AM, Anurag anurag.phadke@gmail.com wrote:

We added 3 more nodes to our cluster and increased the replication
from 0 to 1. The following WARN msgs are coming in the log files:
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110703]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110520]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110704]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110608]
re-syncing mappings with cluster state for types [[crash_reports]]

This has been going for almost an hour now:
curl -XGET http://localhost:9200/_cluster/health

{"cluster_name":"socorro","status":"red","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":7,"active_primary_shards":457,"active_shards":457,"relocating_shards":1,"initializing_shards":0,"unassigned_shards":8}

returns red.

curl -XGET
'http://localhost:9200/_cluster/state?pretty=true&filter_nodes=true&filter_indices=true'
returns everything in:
"state" : "STARTED",

any idea what might be going wrong here?

-anurag

kimchy · July 26, 2011, 2:04pm

Can you do a GET Mapping on the mentioned index and type and gist it?
Basically, its a process that goes and tries to resync mappings across
nodes, and it seems to always think that it needs to resync it...

On Tue, Jul 26, 2011 at 4:46 PM, Anurag anurag.phadke@gmail.com wrote:

yes - 0.17.1

On Mon, Jul 25, 2011 at 9:58 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

Are you using the same version across the cluster?

On Tue, Jul 26, 2011 at 4:41 AM, Anurag anurag.phadke@gmail.com wrote:

We added 3 more nodes to our cluster and increased the replication
from 0 to 1. The following WARN msgs are coming in the log files:
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110703]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110520]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110704]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110608]
re-syncing mappings with cluster state for types [[crash_reports]]

This has been going for almost an hour now:
curl -XGET http://localhost:9200/_cluster/health

{"cluster_name":"socorro","status":"red","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":7,"active_primary_shards":457,"active_shards":457,"relocating_shards":1,"initializing_shards":0,"unassigned_shards":8}

returns red.

curl -XGET
'
http://localhost:9200/_cluster/state?pretty=true&filter_nodes=true&filter_indices=true
'
returns everything in:
"state" : "STARTED",

any idea what might be going wrong here?

-anurag

Anurag_Phadke · July 26, 2011, 2:21pm

here u go: gist:e36e0e0a1140d0adcaa8 · GitHub
seems to be some issue wrt date field.
default mapping file (socorro_mapping.json):
gist:8f145b1c60d1f6a47aa3 · GitHub

On Tue, Jul 26, 2011 at 7:04 AM, Shay Banon
shay.banon@elasticsearch.com wrote:

Can you do a GET Mapping on the mentioned index and type and gist it?
Basically, its a process that goes and tries to resync mappings across
nodes, and it seems to always think that it needs to resync it...

On Tue, Jul 26, 2011 at 4:46 PM, Anurag anurag.phadke@gmail.com wrote:

yes - 0.17.1

On Mon, Jul 25, 2011 at 9:58 PM, Shay Banon
shay.banon@elasticsearch.com wrote:

Are you using the same version across the cluster?

On Tue, Jul 26, 2011 at 4:41 AM, Anurag anurag.phadke@gmail.com wrote:

We added 3 more nodes to our cluster and increased the replication
from 0 to 1. The following WARN msgs are coming in the log files:
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110703]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:16 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110520]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110704]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110609]
re-syncing mappings with cluster state for types [[crash_reports]]
11/07/25 18:40:17 WARN cluster.metadata: [Carrion] [socorro_110608]
re-syncing mappings with cluster state for types [[crash_reports]]

This has been going for almost an hour now:
curl -XGET http://localhost:9200/_cluster/health

{"cluster_name":"socorro","status":"red","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":7,"active_primary_shards":457,"active_shards":457,"relocating_shards":1,"initializing_shards":0,"unassigned_shards":8}

returns red.

curl -XGET

'http://localhost:9200/_cluster/state?pretty=true&filter_nodes=true&filter_indices=true'
returns everything in:
"state" : "STARTED",

any idea what might be going wrong here?

-anurag