Found the issue. The monitoring machine log was filled with these:
[2014-02-20 14:05:07,137][DEBUG][action.search.type ]
[elasticsearch.marvel.1.traackr.com] [.marvel-2014.02.20][0],
node[SwW74zsHRSGZd7OIpukWVQ], [P], s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@516c99db]
org.elasticsearch.search.SearchParseException: [.marvel-2014.02.20][0]:
query[ConstantScore(BooleanFilter(+no_cache(@timestamp:[1392919507058 TO
1392923107136]) +cache(_type:index_stats) +cache(@timestamp:[1392922500000
TO 1392923160000])))],from[-1],size[0]: Parse Failure [Failed to parse
source
[{"size":0,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1392919507058,"to":"now"}}},{"term":{"_type":"index_stats"}},{"range":{"@timestamp":{"from":"now-10m/m","to":"now/m"}}}]}}}},"facets":{"timestamp":{"terms_stats":{"key_field":"index.raw","value_field":"@timestamp","order":"term","size":2000}},"primaries.docs.count":{"terms_stats":{"key_field":"index.raw","value_field":"primaries.docs.count","order":"term","size":2000}},"primaries.indexing.index_total":{"terms_stats":{"key_field":"index.raw","value_field":"primaries.indexing.index_total","order":"term","size":2000}},"total.search.query_total":{"terms_stats":{"key_field":"index.raw","value_field":"total.search.query_total","order":"term","size":2000}},"total.merges.total_size_in_bytes":{"terms_stats":{"key_field":"index.raw","value_field":"total.merges.total_size_in_bytes","order":"term","size":2000}},"total.fielddata.memory_size_in_bytes":{"terms_stats":{"key_field":"index.raw","value_field":"total.fielddata.memory_size_in_bytes","order":"term","size":2000}}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:581)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:484)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:469)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:462)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
at
org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:70)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [timestamp]: failed to find mapping for index.raw
at
org.elasticsearch.search.facet.termsstats.TermsStatsFacetParser.parse(TermsStatsFacetParser.java:119)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:94)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:569)
... 12 more
[2014-02-20 14:05:07,137][DEBUG][action.search.type ]
[elasticsearch.marvel.1.traackr.com] All shards failed for phase: [query]
Looking at "failed to find mapping" made me think of the marvel mappings.
Sure enough, I did overwrite the settings but I messed up: instead of
re-naming my custom template I accidentally named it "marvel" essentially
clobbering the default template. No wonder it wasn't able to index the
fields properly.
I restored the template and it all works fine now.
Should have looked at the logs first. Thanks for point that out.
On Thursday, February 20, 2014 1:31:00 PM UTC-5, Boaz Leskes wrote:
Hi George,
I'm wondering why these search request return 400 bad request. My guess is
that it misses data.
Can you look at the ES logs on you server and see whether you have errors?
Also please run the following (note that it doesn't have a body, that's
intentional):
curl http://...:9200/.marvel-2014.02.20/_search?search_type=count
Does this return correctly?
Cheers,
Boaz
On Thursday, February 20, 2014 6:04:10 PM UTC+1, George Stathis wrote:
- Running 0.90.11
- Marvel monitor machine and the cluster it monitors are separated
- Installed Marvel the day after it was released and it was originally
running fine (all panels filled with data)
Checked the dashboard again today (nothing has changed since it was
installed) and some panels in the dashboard like the node stats and the
indices stats are blank. I tried re-installing marvel on the monitoring
box, cleaned out all marvel-* indexes and even wiped out the data path
directory and let it re-build and the problem persisted.
So I took a look at the JS console and it shows the following errors
(host name redacted):
event.returnValue is deprecated. Please use the standard
event.preventDefault() instead. app.js?r=face658:5
POST
http://<some_internal_host_name_>/.marvel-2014.02.20/search?search_type=count
400 (Bad Request) app.js?r=face658:9
TypeError: Cannot read property 'timestamp' of undefined
at
http://<some_internal_host_name>/plugin/marvel/kibana/app/panels/marvel/stats_table/module.js?r=face658:4:10741
at i
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:458)
at
http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:1014
at Object.f.$eval
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:6963)
at Object.f.$digest
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:5755)
at Object.f.$apply
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:7111)
at f
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:11507)
at r
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:13216)
at XMLHttpRequest.v.onreadystatechange
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:13851)
app.js?r=face658:8
POST
http://<some_internal_host_name>/.marvel-2014.02.20/search?search_type=count
400 (Bad Request) app.js?r=face658:9
TypeError: Cannot read property 'timestamp' of undefined
at
http://<some_internal_host_name>/plugin/marvel/kibana/app/panels/marvel/stats_table/module.js?r=face658:4:10741
at i
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:458)
at
http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:1014
at Object.f.$eval
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:6963)
at Object.f.$digest
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:5755)
at Object.f.$apply
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:7111)
at f
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:11507)
at r
(http://<some_internal_host_name>/plugin/marvel/kibana/app/app.js?r=face658:9:13216)
at XMLHttpRequest.v.onreadystatechange
(http://<some_internal_host_name>/_plugin/marvel/kibana/app/app.js?r=face658:9:13851)
app.js?r=face658:8
Retrying call from Thu Feb 20 2014 11:46:56 GMT-0500 (EST)
module.js?r=face658:4
Retrying call from Thu Feb 20 2014 11:46:56 GMT-0500 (EST)
module.js?r=face658:4
Retrying call from Thu Feb 20 2014 11:47:16 GMT-0500 (EST)
module.js?r=face658:4
Retrying call from Thu Feb 20 2014 11:47:16 GMT-0500 (EST)
module.js?r=face658:4
I don't understand how this error just showed up without having changed
anything on the monitor machine or the cluster it monitors. Any clues are
welcome.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d938fb29-0286-439a-b542-ff87a8cc8ba0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.