Hi Experts,
From past few days there is a problem in our graylog2. Need your help to
resolve the issue
Problem: while we are able to see the all messages, the recent is not
working.
-
On the home page we see this message
o Error! Syntax error in search query or empty index.
-
On the health page, we see this message http://host:port/health
Indices & Shards
Information is not up to date! Seems like you have no Graylog2 node
running.
Informaton last updated at 02.04.2014 - 15:03:52 by
hostname-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba
-
On the gralog2_recent, we see this message *There is no shards
allocated*
curl -X GET '
http://localhost:9200/graylog2_recent/message/_search?from=0&size=10&pretty'
{
"error" : "SearchPhaseExecutionException[Failed to execute phase [query],
total failure; shardFailures {[na][graylog2_recent][0]: No active
shards}{[na][graylog2_recent][1]: No active
shards}{[na][graylog2_recent][2]: No active
shards}{[na][graylog2_recent][3]: No active shards}]",
"status" : 500
-
While other indices are running just fine . all below provides
me the result.
curl -X GET '
http://localhost:9200/graylog2_1/message/_search?from=0&size=10&pretty'
curl -X GET '
http://localhost:9200/graylog2_0/message/_search?from=0&size=10&pretty'
curl -X GET '
http://localhost:9200/twitter/message/_search?from=0&size=10&pretty'
-
Here is the cluster health output
[elastic@dm100opera001 elasticsearch]$ curl -XGET '
http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 11,
"active_shards" : 11,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 11
}[elastic@dm100opera001 elasticsearch]$
-
Here is the shards level status : *graylog2_1*
curl -XGET '
http://localhost:9200/_cluster/health/graylog2_1?level=shards&pretty=true'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"indices" : {
"graylog2_1" : {
"status" : "green",
"number_of_shards" : 4,
"number_of_replicas" : 0,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"shards" : {
"0" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"1" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"2" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"3" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
}
}
}
-
Here is the shards level status : *graylog2_0*
curl -XGET '
http://localhost:9200/_cluster/health/graylog2_0?level=shards&pretty=true'
{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 2,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"indices" : {
"graylog2_0" : {
-
"status" : "red",* "number_of_shards" : 4, "number_of_replicas" : 0, "active_primary_shards" : 2, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "shards" : { "0" : { -
"status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "1" : { -
"status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "2" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 }, "3" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } }}
}
-
Here is the shards level status : *graylog2_recent*
curl -XGET '
http://localhost:9200/_cluster/health/graylog2_recent?level=shards&pretty=true
'
{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4,
"indices" : {
"graylog2_recent" : {
-
"status" : "red",* "number_of_shards" : 4, "number_of_replicas" : 0, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 4, "shards" : { "0" : { -
"status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "1" : { "*status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "2" : { "*status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 }, "3" : { -
"status" : "red",* "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } }}
}
-
Here is what we get from the graylog2 logs :
Nodes:
[graylog2-server][Q8PCsFLtTyCAdjCyVKKeEg][inet[/131.177.54.227:9350]]{client=true,
data=false, master=false}, local
[graylog2-node-][w6NEUpNwRxugT6gFiu9IVw][inet[/131.177.54.227:9300]],
master
routing_table:
-- index [graylog2_recent]
----shard_id [graylog2_recent][0]
--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]
----shard_id [graylog2_recent][1]
--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]
----shard_id [graylog2_recent][2]
--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]
----shard_id [graylog2_recent][3]
--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]
-- index [twitter]
----shard_id [twitter][0]
--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][0], node[null], [R], s[UNASSIGNED]
----shard_id [twitter][1]
--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][1], node[null], [R], s[UNASSIGNED]
----shard_id [twitter][2]
--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][2], node[null], [R], s[UNASSIGNED]
----shard_id [twitter][3]
--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][3], node[null], [R], s[UNASSIGNED]
----shard_id [twitter][4]
--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][4], node[null], [R], s[UNASSIGNED]
-- index [graylog2_0]
----shard_id [graylog2_0][0]
--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]
----shard_id [graylog2_0][1]
--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]
----shard_id [graylog2_0][2]
--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
----shard_id [graylog2_0][3]
--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
-- index [graylog2_1]
----shard_id [graylog2_1][0]
--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
----shard_id [graylog2_1][1]
--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
----shard_id [graylog2_1][2]
--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
----shard_id [graylog2_1][3]
--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
routing_nodes:
-----node_id[w6NEUpNwRxugT6gFiu9IVw][V]
--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]
---- unassigned
--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]
--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]
--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]
--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]
--------[twitter][0], node[null], [R], s[UNASSIGNED]
--------[twitter][1], node[null], [R], s[UNASSIGNED]
--------[twitter][2], node[null], [R], s[UNASSIGNED]
--------[twitter][3], node[null], [R], s[UNASSIGNED]
--------[twitter][4], node[null], [R], s[UNASSIGNED]
--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]
--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.