Graylog2 recent is not working. cluster state is red


(Sachin Kumar Chauhan) #1

Hi Experts,
From past few days there is a problem in our graylog2. Need your help to
resolve the issue

Problem: while we are able to see the all messages, the recent is not
working.

  •      On the home page we see this message 
    

o Error! Syntax error in search query or empty index.

  •       On the health page, we see this message http://host:port/health
    

Indices & Shards

Information is not up to date! Seems like you have no Graylog2 node
running.
Informaton last updated at 02.04.2014 - 15:03:52 by
hostname-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba

  •      On the gralog2_recent, we see this message *There is no shards 
    

allocated*

curl -X GET '
http://localhost:9200/graylog2_recent/message/_search?from=0&size=10&pretty'

{

"error" : "SearchPhaseExecutionException[Failed to execute phase [query],
total failure; shardFailures {[na][graylog2_recent][0]: No active
shards}{[na][graylog2_recent][1]: No active
shards}{[na][graylog2_recent][2]: No active
shards}{[na][graylog2_recent][3]: No active shards}]",

"status" : 500

  •      While other indices are running just fine . all below provides 
    

me the result.

                            curl -X GET '

http://localhost:9200/graylog2_1/message/_search?from=0&size=10&pretty'

                            curl -X GET '

http://localhost:9200/graylog2_0/message/_search?from=0&size=10&pretty'

                            curl -X GET '

http://localhost:9200/twitter/message/_search?from=0&size=10&pretty'

  •      Here is the cluster health output
    

[elastic@dm100opera001 elasticsearch]$ curl -XGET '
http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 11,

"active_shards" : 11,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 11

}[elastic@dm100opera001 elasticsearch]$

  •      Here is the shards level status : *graylog2_1*
    

curl -XGET '
http://localhost:9200/_cluster/health/graylog2_1?level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 4,

"active_shards" : 4,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"indices" : {

"graylog2_1" : {

  "status" : "green",

  "number_of_shards" : 4,

  "number_of_replicas" : 0,

  "active_primary_shards" : 4,

  "active_shards" : 4,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0,

  "shards" : {

    "0" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "1" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "2" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "3" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    }

  }

}

}

  •      Here is the shards level status : *graylog2_0*
    

curl -XGET '
http://localhost:9200/_cluster/health/graylog2_0?level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 2,

"active_shards" : 2,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 2,

"indices" : {

"graylog2_0" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 2,
    
    "active_shards" : 2,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 2,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      },
    
      "3" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      }
    
    }
    

    }

    }

  •      Here is the shards level status : *graylog2_recent*
    

curl -XGET '
http://localhost:9200/_cluster/health/graylog2_recent?level=shards&pretty=true
'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 4,

"indices" : {

"graylog2_recent" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 0,
    
    "active_shards" : 0,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 4,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "3" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      }
    
    }
    

    }

    }

  •      Here is what we get from the graylog2 logs :
    

Nodes:

[graylog2-server][Q8PCsFLtTyCAdjCyVKKeEg][inet[/131.177.54.227:9350]]{client=true,
data=false, master=false}, local

[graylog2-node-][w6NEUpNwRxugT6gFiu9IVw][inet[/131.177.54.227:9300]],
master

routing_table:

-- index [graylog2_recent]

----shard_id [graylog2_recent][0]

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][1]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][2]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][3]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

-- index [twitter]

----shard_id [twitter][0]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][1]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][2]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][3]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][4]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

-- index [graylog2_0]

----shard_id [graylog2_0][0]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][1]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][2]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_0][3]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

-- index [graylog2_1]

----shard_id [graylog2_1][0]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][1]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][2]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][3]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

routing_nodes:

-----node_id[w6NEUpNwRxugT6gFiu9IVw][V]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

---- unassigned

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Your cluster is in a red state, which means you have unassigned primary
shards.

Install an ES plugin like elastichq or kopf to give you an idea on what is
happening.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 17 April 2014 15:52, Sachin Kumar Chauhan chauhan.sachinkr@gmail.comwrote:

Hi Experts,
From past few days there is a problem in our graylog2. Need your help
to resolve the issue

Problem: while we are able to see the all messages, the recent is not
working.

  •      On the home page we see this message
    

o Error! Syntax error in search query or empty index.

  •       On the health page, we see this message
    

http://host:port/health

Indices & Shards

Information is not up to date! Seems like you have no Graylog2 node
running.
Informaton last updated at 02.04.2014 - 15:03:52 by
hostname-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba

  •      On the gralog2_recent, we see this message *There is no
    

shards allocated*

curl -X GET 'http://localhost:9200/graylog2_recent/message/_
search?from=0&size=10&pretty'

{

"error" : "SearchPhaseExecutionException[Failed to execute phase [query],
total failure; shardFailures {[na][graylog2_recent][0]: No active
shards}{[na][graylog2_recent][1]: No active shards}{[na][graylog2_recent][2]:
No active shards}{[na][graylog2_recent][3]: No active shards}]",

"status" : 500

  •      While other indices are running just fine . all below
    

provides me the result.

                            curl -X GET 'http://localhost:9200/

graylog2_1/message/_search?from=0&size=10&pretty'

                            curl -X GET 'http://localhost:9200/

graylog2_0/message/_search?from=0&size=10&pretty'

                            curl -X GET 'http://localhost:9200/

twitter/message/_search?from=0&size=10&pretty'

  •      Here is the cluster health output
    

[elastic@dm100opera001 elasticsearch]$ curl -XGET 'http://localhost:9200/_
cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 11,

"active_shards" : 11,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 11

}[elastic@dm100opera001 elasticsearch]$

  •      Here is the shards level status : *graylog2_1*
    

curl -XGET 'http://localhost:9200/_cluster/health/graylog2_1?
level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 4,

"active_shards" : 4,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"indices" : {

"graylog2_1" : {

  "status" : "green",

  "number_of_shards" : 4,

  "number_of_replicas" : 0,

  "active_primary_shards" : 4,

  "active_shards" : 4,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0,

  "shards" : {

    "0" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "1" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "2" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "3" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    }

  }

}

}

  •      Here is the shards level status : *graylog2_0*
    

curl -XGET 'http://localhost:9200/_cluster/health/graylog2_0?
level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 2,

"active_shards" : 2,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 2,

"indices" : {

"graylog2_0" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 2,
    
    "active_shards" : 2,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 2,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      },
    
      "3" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      }
    
    }
    

    }

    }

  •      Here is the shards level status : *graylog2_recent*
    

curl -XGET 'http://localhost:9200/cluster/health/graylog2
recent?level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 4,

"indices" : {

"graylog2_recent" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 0,
    
    "active_shards" : 0,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 4,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "3" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      }
    
    }
    

    }

    }

  •      Here is what we get from the graylog2 logs :
    

Nodes:

[graylog2-server][Q8PCsFLtTyCAdjCyVKKeEg][inet[/131.177.54.227:9350
]]{client=true, data=false, master=false}, local

[graylog2-node-][w6NEUpNwRxugT6gFiu9IVw][inet[/
131.177.54.227:9300]], master

routing_table:

-- index [graylog2_recent]

----shard_id [graylog2_recent][0]

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][1]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][2]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][3]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

-- index [twitter]

----shard_id [twitter][0]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][1]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][2]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][3]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][4]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

-- index [graylog2_0]

----shard_id [graylog2_0][0]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][1]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][2]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_0][3]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

-- index [graylog2_1]

----shard_id [graylog2_1][0]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][1]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][2]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][3]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

routing_nodes:

-----node_id[w6NEUpNwRxugT6gFiu9IVw][V]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

---- unassigned

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z330FE1ydeZ_8E8k3CF9kwPpp2rK03QaoBEBeyTP-HBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Sachin Kumar Chauhan) #3

Hi,
Could this somehow screw up things. my origional host is host100 but
somehow there was another host *host900 did something. *This was a *temp
*copy of host1oo though.

17.04.2014 - 12:07:24 (about 3 hours ago)
host100.masked.masked-384f7259-da7a-4875-81c7-58202d40fae3
org.graylog2.plugin.GraylogServer Started up.
17.04.2014 - 12:03:49 (about 4 hours ago)
host100.masked.masked-7e443036-1178-47dd-ae6b-323997eff7cc
org.graylog2.plugin.GraylogServer Started up.
17.04.2014 - 11:58:44 (about 4 hours ago)
host100.masked.masked-25009db9-0958-4bbb-82e1-908b5e8d3ff5
org.graylog2.plugin.GraylogServer Shutting down.
17.04.2014 - 07:50:20 (about 8 hours ago)
host100.masked.masked-25009db9-0958-4bbb-82e1-908b5e8d3ff5
org.graylog2.plugin.GraylogServer Started up.
15.04.2014 - 00:19:59 (3 days ago)
host100.masked.masked-2a731605-7188-4d70-b8f3-37b220d83c9b
org.graylog2.plugin.GraylogServer Started up.
10.04.2014 - 06:10:51 (7 days ago)
host100.masked.masked-bd4fc6ae-f3d5-4980-b75f-cd821bf21307
org.graylog2.plugin.GraylogServer Started up.
09.04.2014 - 13:37:57 (8 days ago)
host100.masked.masked-aac46ceb-a517-4ca2-910c-cfe5e74f199d
org.graylog2.plugin.GraylogServer Started up.
08.04.2014 - 15:54:10 (9 days ago)
host100.masked.masked-5eea5c4a-3aba-4f45-8c73-b729e8921ee9
org.graylog2.plugin.GraylogServer Started up.
08.04.2014 - 14:15:52 (9 days ago)
host100.masked.masked-ac4fa804-3a83-42c4-aaf5-84b9fddf2583
org.graylog2.plugin.GraylogServer Started up.
03.04.2014 - 14:22:26 (14 days ago)
host100.masked.masked-7391ca14-b7b3-450c-a361-834068f2ab6d
org.graylog2.plugin.GraylogServer Started up.
01.04.2014 - 18:23:37 (16 days ago)
host100.masked.masked-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba
org.graylog2.plugin.GraylogServer Started up.
31.03.2014 - 18:04:37 (17 days ago)
host900.masked.masked-b0f97343-bdc9-4f71-8077-f4e56531a4da
org.graylog2.plugin.GraylogServer Started up.

10.03.2014 - 15:54:40 (about 1 month ago)
host100-6b45d3fd-e4f7-4430-880c-34dd0013d52d
org.graylog2.indexer.Deflector Cycled deflector from <graylog2_0> to
<graylog2_1>

On Thu, Apr 17, 2014 at 11:57 AM, Mark Walkom markw@campaignmonitor.comwrote:

Your cluster is in a red state, which means you have unassigned primary
shards.

Install an ES plugin like elastichq or kopf to give you an idea on what is
happening.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 17 April 2014 15:52, Sachin Kumar Chauhan chauhan.sachinkr@gmail.comwrote:

Hi Experts,
From past few days there is a problem in our graylog2. Need your help
to resolve the issue

Problem: while we are able to see the all messages, the recent is not
working.

  •      On the home page we see this message
    

o Error! Syntax error in search query or empty index.

  •       On the health page, we see this message
    

http://host:port/health

Indices & Shards

Information is not up to date! Seems like you have no Graylog2 node
running.
Informaton last updated at 02.04.2014 - 15:03:52 by
hostname-47d7bbb1-63af-4b9c-b15c-f7d5f6bf89ba

  •      On the gralog2_recent, we see this message *There is no
    

shards allocated*

curl -X GET 'http://localhost:9200/graylog2_recent/message/_
search?from=0&size=10&pretty'

{

"error" : "SearchPhaseExecutionException[Failed to execute phase
[query], total failure; shardFailures {[na][graylog2_recent][0]: No
active shards}{[na][graylog2_recent][1]: No active
shards}{[na][graylog2_recent][2]: No active shards}{[na][graylog2_recent][3]:
No active shards}]",

"status" : 500

  •      While other indices are running just fine . all below
    

provides me the result.

                            curl -X GET 'http://localhost:9200/

graylog2_1/message/_search?from=0&size=10&pretty'

                            curl -X GET 'http://localhost:9200/

graylog2_0/message/_search?from=0&size=10&pretty'

                            curl -X GET 'http://localhost:9200/

twitter/message/_search?from=0&size=10&pretty'

  •      Here is the cluster health output
    

[elastic@dm100opera001 elasticsearch]$ curl -XGET '
http://localhost:9200/_cluster/health?pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 11,

"active_shards" : 11,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 11

}[elastic@dm100opera001 elasticsearch]$

  •      Here is the shards level status : *graylog2_1*
    

curl -XGET 'http://localhost:9200/_cluster/health/graylog2_1?
level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 4,

"active_shards" : 4,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"indices" : {

"graylog2_1" : {

  "status" : "green",

  "number_of_shards" : 4,

  "number_of_replicas" : 0,

  "active_primary_shards" : 4,

  "active_shards" : 4,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0,

  "shards" : {

    "0" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "1" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "2" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    },

    "3" : {

      "status" : "green",

      "primary_active" : true,

      "active_shards" : 1,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 0

    }

  }

}

}

  •      Here is the shards level status : *graylog2_0*
    

curl -XGET 'http://localhost:9200/_cluster/health/graylog2_0?
level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 2,

"active_shards" : 2,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 2,

"indices" : {

"graylog2_0" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 2,
    
    "active_shards" : 2,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 2,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      },
    
      "3" : {
    
        "status" : "green",
    
        "primary_active" : true,
    
        "active_shards" : 1,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 0
    
      }
    
    }
    

    }

    }

  •      Here is the shards level status : *graylog2_recent*
    

curl -XGET 'http://localhost:9200/cluster/health/graylog2
recent?level=shards&pretty=true'

{

"cluster_name" : "graylog2",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 1,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 4,

"indices" : {

"graylog2_recent" : {
  •  "status" : "red",*
    
    "number_of_shards" : 4,
    
    "number_of_replicas" : 0,
    
    "active_primary_shards" : 0,
    
    "active_shards" : 0,
    
    "relocating_shards" : 0,
    
    "initializing_shards" : 0,
    
    "unassigned_shards" : 4,
    
    "shards" : {
    
      "0" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "1" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "2" : {
    
        "*status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      },
    
      "3" : {
    
  •      "status" : "red",*
    
        "primary_active" : false,
    
        "active_shards" : 0,
    
        "relocating_shards" : 0,
    
        "initializing_shards" : 0,
    
        "unassigned_shards" : 1
    
      }
    
    }
    

    }

    }

  •      Here is what we get from the graylog2 logs :
    

Nodes:

[graylog2-server][Q8PCsFLtTyCAdjCyVKKeEg][inet[/131.177.54.227:9350
]]{client=true, data=false, master=false}, local

[graylog2-node-][w6NEUpNwRxugT6gFiu9IVw][inet[/
131.177.54.227:9300]], master

routing_table:

-- index [graylog2_recent]

----shard_id [graylog2_recent][0]

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][1]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][2]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_recent][3]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

-- index [twitter]

----shard_id [twitter][0]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][1]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][2]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][3]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

----shard_id [twitter][4]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

-- index [graylog2_0]

----shard_id [graylog2_0][0]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][1]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

----shard_id [graylog2_0][2]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_0][3]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

-- index [graylog2_1]

----shard_id [graylog2_1][0]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][1]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][2]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

----shard_id [graylog2_1][3]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

routing_nodes:

-----node_id[w6NEUpNwRxugT6gFiu9IVw][V]

--------[twitter][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[twitter][4], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_0][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][0], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][1], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][2], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

--------[graylog2_1][3], node[w6NEUpNwRxugT6gFiu9IVw], [P], s[STARTED]

---- unassigned

--------[graylog2_recent][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][1], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][2], node[null], [P], s[UNASSIGNED]

--------[graylog2_recent][3], node[null], [P], s[UNASSIGNED]

--------[twitter][0], node[null], [R], s[UNASSIGNED]

--------[twitter][1], node[null], [R], s[UNASSIGNED]

--------[twitter][2], node[null], [R], s[UNASSIGNED]

--------[twitter][3], node[null], [R], s[UNASSIGNED]

--------[twitter][4], node[null], [R], s[UNASSIGNED]

--------[graylog2_0][0], node[null], [P], s[UNASSIGNED]

--------[graylog2_0][1], node[null], [P], s[UNASSIGNED]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c194c8fc-f38e-47e0-8261-e9e87abf9fde%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/54g42AgotSI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624Z330FE1ydeZ_8E8k3CF9kwPpp2rK03QaoBEBeyTP-HBA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624Z330FE1ydeZ_8E8k3CF9kwPpp2rK03QaoBEBeyTP-HBA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Thanks & Regards
Sachin Chauhan
9711704060

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANDgdhqxERLBpV6dJTnJnVy-v8CaGC6FqQ%3DhgNivPiHrnwW1cA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4