Elasticsearch and Kibana not communicating

Elastic Stack Elasticsearch
1

Our Elastic Searcnd Kibana has suddenly stopped working after two new servers were added to send winlog events. On start up the kibana status is red for a long time saying "Elasticsearch is still initialising the kibana index" and eventually it times out with "request time out after 30000ms".
After a few minutes it will load but no data will be in there from the servers. CURL XGET displays this:

sudo curl -XGET "http://192.168.60.90:9200/_cluster/health?pretty"{ "cluster_name" : "elasticsearch", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 2946, "active_shards" : 2946, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 3016, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 49.41294867494129}

Elasticsearch.log displays the following:

Would it be the amount of information that is being sent from ES to KB?

2

I would first try to fix your cluster. Your cluster state is red. You have just one node with almost 3000 active shards and even more unassigned shards. How many indexes do you have? Based on the log this is 597. And how are these indexes configured? How many shards, how many replicas? It looks like the default which is 5 shards and 1 replica. Not all primary shards are allocated, it should have 5 times 597 = 2985 active shards. Which means you are missing 39 shards. I also see you have just 1 Gb of memory, also the default I guess. I would start with more memory and see if the cluster can become yellow. And if you have only one node, disable replicas. Also check if you need 5 shards, what is the size of your shards?

A lot of info and questions, but in short, I think your cluster is not stable. First fix you cluster, then start Kibana again.

3

To be honest, I'm knew to this and I have left everything to pretty much default. We have roughly 150 servers which needs winlogbeat installed so thats alot of data. In terms of indexes and nodes, I don't know how many. Would the first step be to increase the memory?

4

If you are using RedHat stop your ES node first. After that

export ES_HEAP_SIZE=4g It will increase to Elastic search heap to 4Gb. As per this half of the total memory is recommended. Heap: Sizing and Swapping | Elasticsearch: The Definitive Guide [2.x] | Elastic

curl -XPUT 'localhost:9200/<your_index_name>/_settings' -d ' { "index" : { "number_of_replicas" : 0 } }'

It will disable the replicas

Then start your node.

Note: This heap setting works only for current terminal. Hope this will fix your cluster

6

I assumed you are using your winlog beat template. Replace this with your template and try.

Note: Works only for new indeces
{ "mappings": { "_default_": { "_all": { "enabled": true, "norms": { "enabled": false } }, "dynamic_templates": [ { "template1": { "mapping": { "doc_values": true, "ignore_above": 1024, "index": "not_analyzed", "type": "{dynamic_type}" }, "match": "*" } } ], "properties": { "@timestamp": { "type": "date" }, "message": { "index": "analyzed", "type": "string" } } } }, "settings": { "index.refresh_interval": "5s", "number_of_replicas": 0 }, "template": "winlogbeat-*" }

To modify the replicas for old one use

curl -XPUT 'localhost:9200/winlogbeat_*/_settings' -d '{"index" : {"number_of_replicas" : 0}}'

If you dont have curl available in windows please install and try.

7

I did that but i get the following: http://www.pastebin.com/EHDYprQv
I replaced the template like you suggested but when I go to http://192.168.60.90:9200/winlogbeat?pretty I just get:
{
"winlogbeat" : {
"aliases" : { },
"mappings" : { },
"settings" : {
"index" : {
"creation_date" : "1468851642631",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "2Eb-f_L8RGS0tUSuv1NrMA",
"version" : {
"created" : "2030199"
}
}
},
"warmers" : { }
}

so the replica hasn't changed.

8

please edit the link pastbin should be pastebin

9

Replace the "_" with a "-", the error mentions an unknown index, if you look closely you can see the names of the indexes are slightly different.