Hi again,
Yep, I had added the required settings to the yaml files first.
I tried the steps you described above, and it did not have any effect --
still no template present, and still getting the error. Since it wasn't
too much trouble, I started over from scratch -- rebuilt the monitoring
cluster, but also uninstalled then reinstalled the plugin in the production
cluster, and restarted. After this, I saw a bunch of update_mapping calls.
The template was present, and the errors went away. It seems that some
state regarding Marvel is kept in the production cluster, and whatever it
was got cleared when I reinstalled the plugin there. That may be something
worth mentioning in the installation docs.
In any case, thanks for your help -- it's all working now!
Cheers,
Ross
On Tuesday, 28 October 2014 09:48:12 UTC+11, Boaz Leskes wrote:
Hey,
You probably did but just double checking- did you change the settings in
the yaml files before restarting the nodes?There is an easier way to fix this than a full restart: first restart a
single node on production. That will cause the agent to check again for the
template. Verify that the template was added. The delete all .marvel-2014*
indices on the monitoring cluster and let them be recreated base on the
template.Boaz
—
Sent from Mailbox https://www.dropbox.com/mailboxOn Mon, Oct 27, 2014 at 11:25 PM, Ross Simpson <simp...@gmail.com
<javascript:>> wrote:Hi Boaz,
To install, I ran
bin/plugin --install elasticsearch/marvel/latest
on each node in both clusters, then restarted both clusters.
Since then, I have tried several things, including deleting the indexes
from the monitoring cluster and reinstalling the plugin on the monitoring
cluster. I'll try now to delete all the marvel indexes, uninstall, then
reinstall marvel into both clusters.I'm a bit stumped otherwise, so I'm all ears for any other suggestions.
Cheers,
RossOn Tuesday, 28 October 2014 08:30:54 UTC+11, Boaz Leskes wrote:
It looks like something is wrong is indeed wrong with your marvel index
template which should be there before data is indexed. How did you install
marvel? Did you perhaps delete the data folder of the monitoring cluster
after production was already shipping data?Cheers,
BoazOn Monday, October 27, 2014 7:45:34 AM UTC+1, Ross Simpson wrote:
To troubleshoot a little more, I rebuilt the monitoring cluster to use
Elasticsearch 1.1.1, which matches the ES version used in the production
cluster. No luck.On the Overview dashboard, I can see some data (summary, doc count,
search and indexing rates are all populated [screenshot attached]), but but
both the nodes and indices sections are empty other than the errors
mentioned in the previous post. Cluster pulse doesn't show any events at
all; node stats and index stats do both show data.Any further suggestions would be greatly appreciated
Cheers,
RossOn Monday, 27 October 2014 11:15:42 UTC+11, Ross Simpson wrote:
I've got a brand-new Marvel installation, and am having some
frustrating issues with it: on the overview screen, I am constantly getting
errors like:
Oops! FacetPhaseExecutionException[Facet [timestamp]: failed to
find mapping for node.ip_port.raw]Production cluster:
- Elasticsearch 1.1.1
- Marvel 1.2.1
- Running in vSphere
Monitoring cluster:
- Elasticsearch 1.3.4
- Marvel 1.2.1
- Running in AWS
After installing the plugin and bouncing all nodes in both clusters,
Marvel seems to be working -- an index has been created in the monitoring
cluster (.marvel-2014.10.26), and I see thousands of documents in
there. There are documents with the following types: cluster_state,
cluster_stats, index_stats, indices_stats, node_stats. So, it does
seem that data is being shipped from the prod cluster to the monitoring
cluster.I've seen in the user group that other people have had similar issues.
Some of those mention problems with the marvel index template. I don't
seem to have any at all templates in my monitoring cluster:$ curl -XGET localhost:9200/_template/
{}I tried manually adding the default template (as described in
Elasticsearch Platform — Find real-time answers at scale | Elastic),
but that didn't seem to have any effect.So far, I've seen just two specific errors in Marvel:
- FacetPhaseExecutionException[Facet [timestamp]: failed to find
mapping for node.ip_port.raw]- FacetPhaseExecutionException[Facet [timestamp]: failed to find
mapping for index.raw]I've also looked through the logs on both the production and
monitoring clusters, and the only errors are in the monitoring cluster
resulting from queries from the Marvel UI, like this:[2014-10-27 11:08:13,427][DEBUG][action.search.type ] [ip-10-4-
1-187] [.marvel-2014.10.27][1], node[SR_hriFmTCav-8ofbKU-8g], [R], s[
STARTED]: Failed to execute [org.elasticsearch.action.search.
SearchRequest@661dc47e]
org.elasticsearch.search.SearchParseException: [.marvel-2014.10.27][1
]: query[ConstantScore(BooleanFilter(+: +cache(_type:index_stats) +
cache(@timestamp:[1414367880000 TO 1414368540000])))],from[-1],size[10
]: Parse Failure [Failed to parse source [{"size":10,"query":{
"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{
"match_all":{}},{"term":{"_type":"index_stats"}},{"range":{
"@timestamp":{"from":"now-10m/m","to":"now/m"}}}]}}}},"facets":{
"timestamp":{"terms_stats":{"key_field":"index.raw","value_field":
"@timestamp","order":"term","size":2000}},"primaries.docs.count":{
"terms_stats":{"key_field":"index.raw","value_field":
"primaries.docs.count","order":"term","size":2000}},
"primaries.indexing.index_total":{"terms_stats":{"key_field":
"index.raw","value_field":"primaries.indexing.index_total","order":
"term","size":2000}},"total.search.query_total":{"terms_stats":{
"key_field":"index.raw","value_field":"total.search.query_total",
"order":"term","size":2000}},"total.merges.total_size_in_bytes":{
"terms_stats":{"key_field":"index.raw","value_field":
"total.merges.total_size_in_bytes","order":"term","size":2000}},
"total.fielddata.memory_size_in_bytes":{"terms_stats":{"key_field":
"index.raw","value_field":"total.fielddata.memory_size_in_bytes",
"order":"term","size":2000}}}}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:660)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:516)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:488)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:257)
at org.elasticsearch.search.action.
SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:
206)
at org.elasticsearch.search.action.
SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:
203)
at org.elasticsearch.search.action.
SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:
517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException
: Facet [timestamp]: failed to find mapping for index.raw
at org.elasticsearch.search.facet.termsstats.
TermsStatsFacetParser.parse(TermsStatsFacetParser.java:126)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:644</
...--You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/ormCr0X7QoI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7157c765-2576-4ae3-b83f-3a1753f4d2ec%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7157c765-2576-4ae3-b83f-3a1753f4d2ec%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/415a71cc-f593-4398-b57d-fcfa0fba36f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.