I am new to Elasticsearch, and I had been running Elasticsearch, Kibana, Filebeat, and Zabbix Server.
My visualization stopped on August 28. I can pull visualization before that just fine. But when I try to watch current visualization. Nothing. I am getting the error
Could not index event to ElasticSearch.
Can someone advise, as to the first steps to look into. I assume it has to do with an update that took effect.
Thank you in advance.
Deb
Welcome to our community! 
What do your Elasticsearch logs show?
I am not very confident in what I am looking for here.
But this is what I find with a tail -f command.
[2020-09-28T06:25:13,774][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.5gb[11.9%], replicas will not be assigned to this node
[2020-09-28T06:25:43,836][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.4gb[11.9%], replicas will not be assigned to this node
[2020-09-28T06:26:13,887][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.4gb[11.8%], replicas will not be assigned to this node
[2020-09-28T06:26:43,924][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.3gb[11.8%], replicas will not be assigned to this node
[2020-09-28T06:27:13,981][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.2gb[11.8%], replicas will not be assigned to this node
[2020-09-28T06:27:44,024][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.3gb[11.8%], replicas will not be assigned to this node
[2020-09-28T06:28:14,108][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low disk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elasticsearch/nodes/0] free: 18.2gb[11.7%], replicas will not be assigned to this node
[2020-09-28T06:28:44,147][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] rerouting shards: [one or more nodes has gone under the high or low watermark]
Logstash logfile:
[2020-09-28T12:24:00,594][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x243c01c7], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
[2020-09-28T12:24:00,594][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x2fa820a8], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
[2020-09-28T12:24:00,598][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-archives-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x7c1544b5], :response=>{"index"=>{"_index"=>"wazuh-archives-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
[2020-09-28T12:24:01,502][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x600ede06], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
[2020-09-28T12:24:01,503][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x2826615c], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
[2020-09-28T12:24:01,504][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.28", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x3343c290], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.28", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
Logstash errors on status of logstash running:
[logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09
Sep 28 12:31:34 ElasticSearch logstash[1049]: [2020-09-28T12:31:34,483][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09
What is the output from _cat/nodes?v?
What am I typing wrong?
_cat/nodes?v ?
-bash: _cat/nodes?v: No such file or directory
I ran it with the curl command with no luck, I am not for sure of the port.
curl -X GET "localhost:9200/_cat/nodes?v&pretty"
curl: (7) Failed to connect to localhost port 9200: Connection refused
You need to run it against Elasticsearch. If it's not running on localhost, then change to your IP or DNs entry.
curl -X Get "XXX.XX.XXX.XXX/_cat/nodes?v&h=id,ip,port,v,m&pretty"
I then get no answer
It just sits there.
Sounds like Elasticsearch is having issues then, you may want to check its logs.
[2020-09-28T00:59:12,777][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low di
sk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elastic
search/nodes/0] free: 23.1gb[14.9%], replicas will not be assigned to this node
[2020-09-28T00:59:42,814][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] rerout
ing shards: [one or more nodes has gone under the high or low watermark]
[2020-09-28T01:00:42,915][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low di
sk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elastic
search/nodes/0] free: 23.2gb[14.9%], replicas will not be assigned to this node
[2020-09-28T01:01:12,956][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low di
sk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elastic
search/nodes/0] free: 23.2gb[14.9%], replicas will not be assigned to this node
[2020-09-28T01:01:43,019][INFO ][o.e.c.r.a.DiskThresholdMonitor] [node-1] low di
sk watermark [85%] exceeded on [NfUpB_9USYOP63RF3NIEmA][node-1][/var/lib/elastic
search/nodes/0] free: 23.2gb[14.9%], replicas will not be assigned to this node
this is what I find in the elasticsearch log
Any other thoughts what to check on ? Reinstall Elasticsearch?
When I do a status of elasticsearch, the state is at a degraded status?
Please help.
"name" : "node-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "HgSWRRDZR76gW2a6NJjANg",
"version" : {
"number" : "7.5.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
"build_date" : "2019-12-16T22:57:37.835892Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
thoughts, does Elasticsearch look ok?
It looks like you are running low on disk space according to the logs.
Is it something in production? Or you are just testing?
If the later, could you please restart elasticsearch and share the full logs?
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Here is what is located after a restart of Elasticsearch: (I am still receiving no data on Kibana since 8.30.2020) But I can view data prior to that date.
020-09-29T17:50:47,342][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1]
adding template [wazuh-agent] for index patterns [wazuh-monitoring-3.x-*]
[2020-09-29T17:50:50,533][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster h
ealth status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_ta
sk_manager_1][0]]]).'
I am also receiving this in alert.log on Ubuntu, Wazuh Manager system.
2020 Sep 29 00:12:39 (elasticsearch) any->/var/log/syslog
Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
Sep 29 00:12:29 ElasticSearch logstash[1049]: [2020-09-29T00:12:29,100][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2020.09.29", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x61e265ff>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2020.09.29", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.