Indices problem

Hi,

I think I made a mistake, curator wouldn't work and I went to /elasticeasrch/nodes/0/indices and I deleste all indices !! :confused: Now I can't receive any logs :confused:

If you only have a single node and no replicas, nor any snapshot to restore from, that would indeed be a problem. What is the status of your cluster? Have you restarted it?

[root@frghcslnetv12 elasticsearch]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-06-21 10:59:58 CEST; 41min ago
     Docs: http://www.elastic.co
  Process: 60597 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 60599 (java)
    Tasks: 50
   CGroup: /system.slice/elasticsearch.service
           └─60599 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m...

Jun 21 10:59:58 frghcslnetv12 systemd[1]: Starting Elasticsearch...
Jun 21 10:59:58 frghcslnetv12 systemd[1]: Started Elasticsearch.

Yes I restart it, and I have just one node :confused: !!! PLEASE HELP

If you have deleted all the data from the file system and have no snapshot to restore from, it is lost. Is the node running?

How can I know ?

What is the output of the cluster health API?

[root@frghcslnetv12 elasticsearch]#  curl XGET "http://127.0.0.1:9200/_cluster/health"
curl: (6) Could not resolve host: XGET; Unknown error
{"cluster_name":"graylog","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":4,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@frghcslnetv12 elasticsearch]#

Looks like it is up and running:

{
	"cluster_name": "graylog",
	"status": "green",
	"timed_out": false,
	"number_of_nodes": 1,
	"number_of_data_nodes": 1,
	"active_primary_shards": 4,
	"active_shards": 4,
	"relocating_shards": 0,
	"initializing_shards": 0,
	"unassigned_shards": 0,
	"delayed_unassigned_shards": 0,
	"number_of_pending_tasks": 0,
	"number_of_in_flight_fetch": 0,
	"task_max_waiting_in_queue_millis": 0,
	"active_shards_percent_as_number": 100.0
}

As there are indices in the cluster it may even be receiving data again.

[root@frghcslnetv12 elasticsearch]# df -kh
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vgroot-root     22G  9.6G   12G  46% /
devtmpfs                   3.8G     0  3.8G   0% /dev
tmpfs                      3.9G     0  3.9G   0% /dev/shm
tmpfs                      3.9G   28M  3.8G   1% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                 1014M  208M  807M  21% /boot
/dev/mapper/vgdata-lvdata  196G  265M  188G   1% /data
tmpfs                      781M     0  781M   0% /run/user/0

My path.data is : /data/elasticsearch

And filebeat is sending data but I m not receveing them !!

1% /data !!

What is the output of the cat indices API?

[root@frghcslnetv12 elasticsearch]# curl XGET "http://127.0.0.1:9200/_cat/indices/twi*?v&s=index"
curl: (6) Could not resolve host: XGET; Unknown error
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

What does curl -XGET "http://127.0.0.1:9200/_cat/indices give? What indices do you have in the cluster?

[root@frghcslnetv12 elasticsearch]# curl -XGET "http://127.0.0.1:9200/_cat/indices"
green open graylog_0 XYAgOe9GSeGSMpv6vNEk5w 4 0 347825 0 547.1mb 547.1mb

Looks like Graylog is able to write data. Are you expecting any other indices in the cluster?

The problème is the path.data is /data/elasticsearch

and logs are stored on : / (it will be full in few minutes !! )

So logs are not sent to : Elasticsearch

What does your elasticsearch.yml file look like?

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["172.16.250.30"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

How did you install Elasticsearch? Where was that elasticsearch.yml file located?

cd /etc/elasticsearch/elasticsearch.yml

As far as I can see the file looks fine and is in the correct location, so I am not sure what is wrong. Has anything changed recently?