Kibana discards old records after ES restart

Elastic Stack Kibana
1

Hi, i m a newbie working with kibana and elasticseach and i need some help :slight_smile: i allready searched about my problem but i couldnt figure out how to solve this. @warkolm adviced me to write my own thread :+1:

In my case, i m using rsyslog+ES+Kibana for loggin and everything s fine then I ve restarted ES and kibana asked me to create new index pattern. so i created an index pattern same as the previous pattern. But after this step i could not view any old log records with kibana. I checked the old files from local cmdline they re still exist but kibana doesnt load any of them. i checked my nodes, i ve only one. But i ve some concerns about my elasticsearch.yml file.
could you please guide me to solve this issue. thanks

here s my indices dir
elasticsearch/nodes/0/indices# ls -altrh
total 156K
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Dec 29 15:17 ..
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Dec 29 15:17 .kibana
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Dec 29 15:26 logstash-2015.12.29
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Dec 30 00:17 logstash-2015.12.30
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Dec 31 00:17 logstash-2015.12.31
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 2 06:42 logstash-2016.01.02
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 3 06:35 logstash-2016.01.03
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 4 06:52 logstash-2016.01.04
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 5 06:50 logstash-2016.01.05
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 6 06:54 logstash-2016.01.06
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 7 06:26 logstash-2016.01.07
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 8 06:38 logstash-2016.01.08
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 9 06:47 logstash-2016.01.09
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 10 06:43 logstash-2016.01.10
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 11 06:30 logstash-2016.01.11
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 12 06:36 logstash-2016.01.12
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 13 06:36 logstash-2016.01.13
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 14 00:00 logstash-2016.01.14
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 15 00:01 logstash-2016.01.15
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 16 00:01 logstash-2016.01.16
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 17 00:00 logstash-2016.01.17
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 18 00:00 logstash-2016.01.18
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 19 00:00 logstash-2016.01.19
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 20 00:00 logstash-2016.01.20
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 21 00:03 logstash-2016.01.21
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 22 01:34 logstash-2016.01.22
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 23 00:19 logstash-2016.01.23
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 24 00:28 logstash-2016.01.24
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 25 01:30 logstash-2016.01.25
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 26 00:01 logstash-2016.01.26
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 27 02:00 logstash-2016.01.27
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 28 02:00 logstash-2016.01.28
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 29 02:00 logstash-2016.01.29
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 30 02:00 logstash-2016.01.30
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Jan 31 02:00 logstash-2016.01.31
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Feb 1 05:02 logstash-2016.02.01
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Feb 2 05:40 logstash-2016.02.02
drwxr-xr-x 39 elasticsearch elasticsearch 4.0K Feb 3 06:19 .
drwxr-xr-x 8 elasticsearch elasticsearch 4.0K Feb 3 06:19 logstash-2016.02.03

my nodes dir
/elasticsearch/nodes# ls -altrh
total 12K
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Dec 29 15:17 ..
drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Dec 29 15:17 .
drwxr-xr-x 4 elasticsearch elasticsearch 4.0K Dec 29 15:17 0

and my elasticseach.yml - i uncommented only these two lines in network section all other lines are commented

network.host: localhost
http.port: 9200

my cluster status
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' {
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 11,
"active_shards" : 11,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 11,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}

2

hi there, i ll appreciate if someone help me to solve this :slight_smile:

3

What's your Kibana index pattern? What do you see? Some screenshots would help.

4

Hi @tbragin,

While i am writing a reply to you i check my indices files dir and .yml file so i realized that my indices path is changed after the day i restarted ES. Thats why kibana couldnt view old logs. The point is checking elasticsearch.yml PATHS section if indices files' path is true. So its all about being newbie..

By the way i ve watched your videos about kibana they were very useful thaks and i m glad to take a reply from you @tbragin , as everyone can see just one reply from you is enough to solve a problem :smiley:

1 Like
5

Thank you so much! :slightly_smiling: Glad the issue is resolved. Thank you for following up, and letting us know what the solution is!