.marvel-es-YYYY.MM.DD index deleted HOW-TO create it again

monitoring

(Nikos Skalis) #1

Dear All,

i recently installed Elasticsearch 2.0 and Marvel 2.0
i was experimenting with Curator and I accidentally delete the .marvel-es-YYY.MM.DD index

health status index           pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .kibana           1   1          3            1     26.7kb         26.7kb 
yellow open   .marvel-es-data   1   1          1            0      3.6kb          3.6kb 

i have only 1 configuration change in elasticsearch.yml

marvel.agent.interval: 1m

but i cannot see any statistics on the marvel page because the corresponding index is simple not re-created, even waiting for 1 day to pass i cannot see the index created.
below are some screenshots from Kibana and Marvel.

could you please advise on how to troubleshoot this issue ?


Nikos


(Tanguy) #2

Hi Nikos,

Did you configure marvel to index data locally or remotely using Http?

Unfortunately I wasn't able to reproduce your issue on my laptop. This is what I tested: I installed both ES 2.0 and Marvel 2.0 with default configuration (ie marvel indexes data in the same cluster being monitored).

After a while, I can see marvel indices corectly created:

GET /_cat/indices?v
health status index                 pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .marvel-es-2015.11.09   1   1          8            0     20.6kb         20.6kb 
yellow open   .marvel-es-data         1   1          1            0      3.2kb          3.2kb 

So I deleted the .marvel-es-YYYY-MM-DD index:

DELETE /.marvel-es-2015.11.09
{
  "acknowledged" : true
}

And the elasticsearch logs reported:

[INFO ][cluster.metadata         ] [General Orwell Taylor] [.marvel-es-2015.11.09] deleting index

After few seconds, the logs show that the index has been re created:

[INFO ][cluster.metadata         ] [General Orwell Taylor] [.marvel-es-2015.11.09] creating index, cause [auto(bulk api)], templates [.marvel-es], shards [1]/[1], mappings [shards, _default_, index_stats, index_recovery, cluster_state, cluster_info, cluster_stats, node_stats, indices_stats]
[2015-11-09 11:31:36,911][INFO ][cluster.metadata         ] [General Orwell Taylor] [.marvel-es-2015.11.09] update_mapping [node_stats]
[2015-11-09 11:31:36,928][INFO ][cluster.metadata         ] [General Orwell Taylor] [.marvel-es-2015.11.09] update_mapping [cluster_stats]

So does GET /_cat/indices?v:

health status index                 pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .marvel-es-2015.11.09   1   1          4            0     20.6kb         20.6kb 
yellow open   .marvel-es-data         1   1          1            0      3.2kb          3.2kb 

I also tested with Marvel configured to export data using HTTP to another elasticsearch cluster and it also works.

Can you please check your logs to see if there is an error that can help us to find the issue?

Thanks

-- Tanguy


(Nikos Skalis) #3

Thank you Tanguy for looking into this.
Here are the logs when I restart elasticsearch

[root@nl-ams02c-ispdat01 ~]# tail -f /var/log/elasticsearch/*.log &
[1] 37618
[root@nl-ams02c-ispdat01 ~]# systemctl stop elasticsearch
[2015-11-09 12:06:06,174][INFO ][node                     ] [nl-ams02c-ispdat01] stopping ...
[2015-11-09 12:06:06,315][INFO ][node                     ] [nl-ams02c-ispdat01] stopped
[2015-11-09 12:06:06,315][INFO ][node                     ] [nl-ams02c-ispdat01] closing ...
[2015-11-09 12:06:06,330][INFO ][node                     ] [nl-ams02c-ispdat01] closed
[root@nl-ams02c-ispdat01 ~]# systemctl start elasticsearch
[root@nl-ams02c-ispdat01 ~]# [2015-11-09 12:06:22,039][INFO ][node                     ] [nl-ams02c-ispdat01] version[2.0.0], pid[37635], build[de54438/2015-10-22T08:09:48Z]
[2015-11-09 12:06:22,039][INFO ][node                     ] [nl-ams02c-ispdat01] initializing ...
[2015-11-09 12:06:22,301][INFO ][plugins                  ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-09 12:06:22,337][INFO ][env                      ] [nl-ams02c-ispdat01] using [1] data paths, mounts [[/usr (/dev/mapper/centos-usr)]], net usable_space [1.2tb], net total_space [1.2tb], spins? [possibly], types [xfs]
[2015-11-09 12:06:23,460][INFO ][marvel.agent.settings    ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated
[2015-11-09 12:06:24,586][INFO ][node                     ] [nl-ams02c-ispdat01] initialized
[2015-11-09 12:06:24,586][INFO ][node                     ] [nl-ams02c-ispdat01] starting ...
[2015-11-09 12:06:24,898][INFO ][transport                ] [nl-ams02c-ispdat01] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2015-11-09 12:06:24,908][INFO ][discovery                ] [nl-ams02c-ispdat01] IP-spotlight/EBST-cZUQa-jGhnnBkhd0g
[2015-11-09 12:06:27,954][INFO ][cluster.service          ] [nl-ams02c-ispdat01] new_master {nl-ams02c-ispdat01}{EBST-cZUQa-jGhnnBkhd0g}{127.0.0.1}{127.0.0.1:9300}{tag=exabgp, group=IP-spotlight}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-11-09 12:06:28,026][WARN ][common.network           ] [nl-ams02c-ispdat01] publish address: {0.0.0.0} is a wildcard address, falling back to first non-loopback: {fe80::569f:35ff:fe0a:99c0}
[2015-11-09 12:06:28,026][INFO ][http                     ] [nl-ams02c-ispdat01] publish_address {[fe80::569f:35ff:fe0a:99c0]:9200}, bound_addresses {[::]:9200}
[2015-11-09 12:06:28,028][INFO ][node                     ] [nl-ams02c-ispdat01] started
[2015-11-09 12:06:28,095][INFO ][license.plugin.core      ] [nl-ams02c-ispdat01] license [b568290d-5957-4ab7-bbd4-58fde74bb508] - valid
[2015-11-09 12:06:28,097][ERROR][license.plugin.core      ] [nl-ams02c-ispdat01] 
#
# License will expire on [Monday, December 07, 2015]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
# 
# Commercial plugins operate with reduced functionality on license expiration:
# - marvel
#  - The agent will stop collecting cluster and indices metrics
[2015-11-09 12:06:28,120][INFO ][gateway                  ] [nl-ams02c-ispdat01] recovered [3] indices into cluster_state

[root@nl-ams02c-ispdat01 ~]# kill -SIGTERM 37618
[1]+  Terminated              tail -f /var/log/elasticsearch/*.log

i deleted also the ".marvel-es-data" index as well, and re-install marvel also, but in vain.

health status index         pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .kibana         1   1          3            0     24.3kb         24.3kb 
green  open   <SNIP!>   1   0     819859            0    131.9mb        131.9mb 
green  open   <SNIP!>   1   0    1921373            0    284.4mb        284.4mb 

thank you a lot for looking into this. i really appreciate it.
i can provide a team viewer session if necessary in order to troubleshoot the issue.

Nikos


(Tanguy) #4

I don't see anything strange in the log.

Can you please set Marvel to debug level? To do that, you need to edit the logging.yml file located in the elasticsearch config directory and add the line marvel: DEBUG under the logger entry:

logger:
  ...
  marvel: DEBUG

Then you can restart and wait few minutes so that Marvel will collect some data (it collects every minute with your config)

Thanks


(Nikos Skalis) #5
[root@nl-ams02c-ispdat01 elasticsearch]# [2015-11-09 13:31:39,729][INFO ][node                     ] [nl-ams02c-ispdat01] version[2.0.0], pid[47891], build[de54438/2015-10-22T08:09:48Z]
[2015-11-09 13:31:39,730][INFO ][node                     ] [nl-ams02c-ispdat01] initializing ...
[2015-11-09 13:31:39,992][INFO ][plugins                  ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-09 13:31:40,026][INFO ][env                      ] [nl-ams02c-ispdat01] using [1] data paths, mounts [[/usr (/dev/mapper/centos-usr)]], net usable_space [1.2tb], net total_space [1.2tb], spins? [possibly], types [xfs]
[2015-11-09 13:31:41,034][INFO ][marvel.agent.settings    ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated
[2015-11-09 13:31:42,181][INFO ][node                     ] [nl-ams02c-ispdat01] initialized
[2015-11-09 13:31:42,182][INFO ][node                     ] [nl-ams02c-ispdat01] starting ...
[2015-11-09 13:31:42,182][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-info-collector]
[2015-11-09 13:31:42,182][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-state-collector]
[2015-11-09 13:31:42,182][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [indices-stats-collector]
[2015-11-09 13:31:42,183][DEBUG][marvel.agent.collector.shards] [nl-ams02c-ispdat01] starting collector [shards-collector]
[2015-11-09 13:31:42,183][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [index-stats-collector]
[2015-11-09 13:31:42,183][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-stats-collector]
[2015-11-09 13:31:42,183][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [index-recovery-collector]
[2015-11-09 13:31:42,183][DEBUG][marvel.agent.collector.node] [nl-ams02c-ispdat01] starting collector [node-stats-collector]
[2015-11-09 13:31:42,478][INFO ][transport                ] [nl-ams02c-ispdat01] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2015-11-09 13:31:42,487][INFO ][discovery                ] [nl-ams02c-ispdat01] IP-spotlight/QYc0hiKnTQ2AbL-0ZtjV3Q
[2015-11-09 13:31:45,532][INFO ][cluster.service          ] [nl-ams02c-ispdat01] new_master {nl-ams02c-ispdat01}{QYc0hiKnTQ2AbL-0ZtjV3Q}{127.0.0.1}{127.0.0.1:9300}{tag=exabgp, group=IP-spotlight}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-11-09 13:31:45,541][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - waiting until gateway has recovered from disk
[2015-11-09 13:31:45,612][WARN ][common.network           ] [nl-ams02c-ispdat01] publish address: {0.0.0.0} is a wildcard address, falling back to first non-loopback: {fe80::569f:35ff:fe0a:99c0}
[2015-11-09 13:31:45,613][INFO ][http                     ] [nl-ams02c-ispdat01] publish_address {[fe80::569f:35ff:fe0a:99c0]:9200}, bound_addresses {[::]:9200}
[2015-11-09 13:31:45,614][INFO ][node                     ] [nl-ams02c-ispdat01] started
[2015-11-09 13:31:45,667][INFO ][license.plugin.core      ] [nl-ams02c-ispdat01] license [SNIP!] - valid
[2015-11-09 13:31:45,669][ERROR][license.plugin.core      ] [nl-ams02c-ispdat01] 
#
# License will expire on [Monday, December 07, 2015]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
# 
# Commercial plugins operate with reduced functionality on license expiration:
# - marvel
#  - The agent will stop collecting cluster and indices metrics
[2015-11-09 13:31:45,672][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 13:31:45,673][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - started!
[2015-11-09 13:31:45,690][INFO ][gateway                  ] [nl-ams02c-ispdat01] recovered [3] indices into cluster_state
[2015-11-09 13:31:46,013][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 13:31:46,116][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 13:31:46,138][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 13:31:46,155][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date

[root@nl-ams02c-ispdat01 elasticsearch]# date
Mon Nov  9 13:32:34 CET 2015
[root@nl-ams02c-ispdat01 elasticsearch]# date
Mon Nov  9 13:34:02 CET 2015

(Nikos Skalis) #6

hmmm.. it seems that it doesn't even try to gather any data if i am not mistaken. any idea why ?


(Tanguy) #7

No... Can you try again but with TRACE level instead of DEBUG?

Hopefully we'll see something...


(Nikos Skalis) #8
[2015-11-09 20:18:46,451][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,451][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - started!
[2015-11-09 20:18:46,470][INFO ][gateway                  ] [nl-ams02c-ispdat01] recovered [3] indices into cluster_state
[2015-11-09 20:18:46,798][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,907][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,926][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,929][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:19:42,961][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] collecting data - collectors [cluster-info-collector,cluster-state-collector,indices-stats-collector,shards-collector,index-stats-collector,cluster-stats-collector,index-recovery-collector,node-stats-collector]
[2015-11-09 20:19:42,962][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-info-collector] - collecting data...
[2015-11-09 20:19:43,009][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-info-collector] collector
[2015-11-09 20:19:43,041][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-state-collector] - collecting data...
[2015-11-09 20:19:43,042][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-state-collector] collector
[2015-11-09 20:19:43,056][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [indices-stats-collector] - collecting data...
[2015-11-09 20:19:43,077][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [indices-stats-collector] collector
[2015-11-09 20:19:43,088][TRACE][marvel.agent.collector.shards] [nl-ams02c-ispdat01] collector [shards-collector] - collecting data...
[2015-11-09 20:19:43,089][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [4] collected docs from [shards-collector] collector
[2015-11-09 20:19:43,091][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [index-stats-collector] - collecting data...
[2015-11-09 20:19:43,095][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [3] collected docs from [index-stats-collector] collector
[2015-11-09 20:19:43,099][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-stats-collector] - collecting data...
[2015-11-09 20:19:43,102][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-stats-collector] collector
[2015-11-09 20:19:43,103][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [index-recovery-collector] - collecting data...
[2015-11-09 20:19:43,104][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [index-recovery-collector] collector
[2015-11-09 20:19:43,114][TRACE][marvel.agent.collector.node] [nl-ams02c-ispdat01] collector [node-stats-collector] - collecting data...
[2015-11-09 20:19:43,131][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [node-stats-collector] collector
[2015-11-09 20:19:43,136][TRACE][marvel.agent.exporter.local] exporter [default_local] - exporting data...

(Nikos Skalis) #9

this pattern of log messages keeps repeating every 1min. now i got even more confused.

[2015-11-09 20:18:46,451][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,451][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - started!
[2015-11-09 20:18:46,470][INFO ][gateway                  ] [nl-ams02c-ispdat01] recovered [3] indices into cluster_state
[2015-11-09 20:18:46,798][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,907][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,926][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:18:46,929][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-09 20:19:42,961][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] collecting data - collectors [cluster-info-collector,cluster-state-collector,indices-stats-collector,shards-collector,index-stats-collector,cluster-stats-collector,index-recovery-collector,node-stats-collector]
[2015-11-09 20:19:42,962][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-info-collector] - collecting data...
[2015-11-09 20:19:43,009][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-info-collector] collector
[2015-11-09 20:19:43,041][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-state-collector] - collecting data...
[2015-11-09 20:19:43,042][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-state-collector] collector
[2015-11-09 20:19:43,056][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [indices-stats-collector] - collecting data...
[2015-11-09 20:19:43,077][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [indices-stats-collector] collector
[2015-11-09 20:19:43,088][TRACE][marvel.agent.collector.shards] [nl-ams02c-ispdat01] collector [shards-collector] - collecting data...
[2015-11-09 20:19:43,089][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [4] collected docs from [shards-collector] collector
[2015-11-09 20:19:43,091][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [index-stats-collector] - collecting data...
[2015-11-09 20:19:43,095][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [3] collected docs from [index-stats-collector] collector
[2015-11-09 20:19:43,099][TRACE][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] collector [cluster-stats-collector] - collecting data...
[2015-11-09 20:19:43,102][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [cluster-stats-collector] collector
[2015-11-09 20:19:43,103][TRACE][marvel.agent.collector.indices] [nl-ams02c-ispdat01] collector [index-recovery-collector] - collecting data...
[2015-11-09 20:19:43,104][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [index-recovery-collector] collector
[2015-11-09 20:19:43,114][TRACE][marvel.agent.collector.node] [nl-ams02c-ispdat01] collector [node-stats-collector] - collecting data...
[2015-11-09 20:19:43,131][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] bulk [all] - adding [1] collected docs from [node-stats-collector] collector
[2015-11-09 20:19:43,136][TRACE][marvel.agent.exporter.local] exporter [default_local] - exporting data...
[2015-11-09 20:20:43,140][TRACE][marvel.agent             ] [nl-ams02c-ispdat01] collecting data - collectors [cluster-info-collector,cluster-state-collector,indices-stats-collector,shards-collector,index-stats-collector,cluster-stats-collector,index-recovery-collector,node-stats-collector]

(Tanguy) #10

According to the trace, it works correctly. Every minute marvel collects a bunch of metrics and exports them locally ie on node nl-ams02c-ispdat01.

A curl -XGET 'http://<nl-ams02c-ispdat01 host>:9200/.marvel-es-2015*/_search should return some documents...


(Nikos Skalis) #11

hmm.. no documents returned. can i ask how to completely delete Marvel ?
i did bin/plugin remove but re-installed it but didn't help..


(Tanguy) #12

To uninstall the marvel plugin, run bin/plugin remove marvel-agent. You'll need to delete the .marvel-es-* indices manually.

Then you can try to reinstall it again to see if it works better. Right now I can't say why it does not index data because everything looks good in traces and I did not manage to reproduce the issue.


(Tanguy) #13

@nskalis Did you manage to uninstall / reinstall marvel?


(Nikos Skalis) #14

hi @tanguy . i had done this in the past. but unfortunately it didn't help.
i'll do it again, but i have to do it during the weekend this time.
thank you for the support. it is much appreciated. i'll let you know.


(Tanguy) #15

Sure. I'd like to understand what's going on in your cluster and be sure that there's no bug around.

In case you reproduce it and if there's not much data in your cluster, do you think you could send me your config/data so that I can debug it locally?

Thanks


(Nikos Skalis) #16

data would be a bit difficult to share. am gathering IP prefixes from 13
route reflectors (and countries). and that is a bit of sensitive data from
technical perspective.
but we can have a team viewer session, i'll pass over control to you, and
like that
you 'll have the setup and the data. and we can look into that together, if
that is OK with you.


(Tanguy) #17

Thanks for the proposal but what is the most helpful for me would be to debug locally and check line by line with source code what's going on.

Lastly, can you - still in DEBUG log level - update the sampling interval from 30 sec to 60 sec:

curl -XPUT localhost:9200/_cluster/settings -d '{"transient" : {"marvel.agent.interval" : "30s" }}'

Updating this setting may trigger index re creation.


(Nikos Skalis) #18

hi @tanguy
i removed all plugins
i removed the .kibana index
i installed the new Kibana 4.2.1, Sense, Marvel

but i do not see the .marvel* indices created
i also applied the transient settings suggested but no luck.

Waiting for Marvel Data
It appears that we have not received any data for this cluster. Marvel data is sent every 10 seconds by the agent plugin; once the data appears this page will automatically redirect to the overview page.

and enabled logging in DEBUG mode, the output is :


(Nikos Skalis) #19

cat /var/log/elasticsearch/*.log | grep -i marvel

- marvel

[2015-11-22 08:57:53,525][INFO ][plugins ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-22 08:57:54,587][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated

- marvel

[2015-11-22 09:00:19,138][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:00:29,236][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:00:49,254][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:01:22,461][INFO ][plugins ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-22 09:01:23,506][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated

- marvel

[2015-11-22 09:01:49,570][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:02:01,173][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:02:42,197][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:03:42,356][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:04:42,527][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:05:43,575][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:06:44,878][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:07:24,157][INFO ][plugins ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-22 09:07:25,204][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-info-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-state-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [indices-stats-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.shards] [nl-ams02c-ispdat01] starting collector [shards-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [index-stats-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-stats-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [index-recovery-collector]
[2015-11-22 09:07:25,730][DEBUG][marvel.agent.collector.node] [nl-ams02c-ispdat01] starting collector [node-stats-collector]
[2015-11-22 09:07:29,099][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - waiting until gateway has recovered from disk

- marvel


(Nikos Skalis) #20

[2015-11-22 09:07:29,209][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-22 09:07:29,209][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - started!
[2015-11-22 09:07:29,454][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-22 09:07:29,574][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-22 09:07:52,969][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:12,558][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 30s] updated
[2015-11-22 09:09:12,559][INFO ][marvel.agent ] [nl-ams02c-ispdat01] sampling interval updated to [30s]
[2015-11-22 09:09:12,559][DEBUG][marvel.agent.exporter.local] local exporter [default_local] - currently installed marvel template version [2.0.0] is up-to-date
[2015-11-22 09:09:28,421][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:31,721][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:48,552][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:50,873][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:52,069][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:09:53,069][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:10:15,640][INFO ][rest.suppressed ] /.marvel-es-data/cluster_info/_search Params: {index=.marvel-es-data, type=cluster_info}
[.marvel-es-data] IndexNotFoundException[no such index]
[2015-11-22 09:10:19,649][DEBUG][marvel.agent ] [nl-ams02c-ispdat01] worker shutdown
[2015-11-22 09:10:19,649][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] stopping collector [cluster-info-collector]
[2015-11-22 09:10:19,650][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] stopping collector [cluster-state-collector]
[2015-11-22 09:10:19,650][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] stopping collector [indices-stats-collector]
[2015-11-22 09:10:19,651][DEBUG][marvel.agent.collector.shards] [nl-ams02c-ispdat01] stopping collector [shards-collector]
[2015-11-22 09:10:19,651][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] stopping collector [index-stats-collector]
[2015-11-22 09:10:19,651][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] stopping collector [cluster-stats-collector]
[2015-11-22 09:10:19,651][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] stopping collector [index-recovery-collector]
[2015-11-22 09:10:19,651][DEBUG][marvel.agent.collector.node] [nl-ams02c-ispdat01] stopping collector [node-stats-collector]
[2015-11-22 09:10:29,226][INFO ][plugins ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-22 09:10:30,282][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-info-collector]
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-state-collector]
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [indices-stats-collector]
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.shards] [nl-ams02c-ispdat01] starting collector [shards-collector]
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.indices] [nl-ams02c-ispdat01] starting collector [index-stats-collector]
[2015-11-22 09:10:30,849][DEBUG][marvel.agent.collector.cluster] [nl-ams02c-ispdat01] starting collector [cluster-stats-collector]