Environment: I'm running ES 0.90.10 on a Windows 7 Pro box (my dev machine).
Steps:
- Add the following to your [es directory]/config/elasticsearch.yml
before installing marvel:
action.auto_create_index: false
index.mapper.dynamic: false
- Install marvel plugin
- Restart your ES cluster
- Navigate to http://localhost:9200/_plugin/marvel
Expected:
Either a working marvel dashboard or a very explicit warning/error message
that marvel failed to create its index.
Bonus: detect the settings above and give specific remedies in the
notification (remove those settings, use the auto_create_index whitelist to
allow marvel to create it's own indices, etc.).
Actual:
- Empty marvel dashboard (page renders just fine, just no data to
display), with a short-lived message at the top of the dashboard: "No
results There were no results because no indices were found that match your
selected time span" - The following error repeating in your ES logs 4 times every second:
[2014-01-29 10:21:11,639][ERROR][marvel.agent.exporter ] error sending
data
java.io.FileNotFoundException: http://localhost:9200/_bulk
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.sendCloseExportingConnection(ESExporter.java:232)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportClusterStats(ESExporter.java:172)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportClusterStats(AgentService.java:214)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:180)
at java.lang.Thread.run(Thread.java:744)
This error is a symptom of marvel not being able to create it's own index
due to action.auto_create_index: false in our elasticsearch.yml
My next step was to turn off the action.auto_create_index: false setting in
elasticsearch.yml and re-start the cluster.
This resulted in marvel being able to create it's index, but still failing
to index any data.
- Same issue with the marvel dashboard (no data, no warnings/errors)
- Different error message in the ES logs 4 times/second:
[2014-01-29 10:24:54,793][DEBUG][action.bulk ] [Princess
Python] [.marvel-2014.01.29][0] failed to execute
bulk item (index) index
{[.marvel-2014.01.29][indices_stats][5eVda-3SRBK2TKmjHzFpBA],
source[{"@timestamp":"2014-01-29T1
5:24:54.786Z","cluster_name":"elasticsearch","_source_node":{"id":"LBDVUL_eRt27pnVCno7NBQ","name":"Princess
Python","tra
nsport_address":"inet[dane.hq.fogcreek.com/10.1.1.167:9300]","ip":"10.1.1.167","host":"dane.hq.fogcreek.com","ip_port":"
10.1.1.167:9300"},"primaries":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_m
illis":2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_mil
lis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total
":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"qu
ery_current":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"cur
rent_size_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"to
tal":19,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"tota
l_time_in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"
fielddata":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory
in_bytes":0}},"total":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_millis":
2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_millis":0,
"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"mi
ssing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"query_cur
rent":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_si
ze_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"total":19
,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"total_time
in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fieldda
ta":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory_in_byt
es":0}}}]}
org.elasticsearch.indices.TypeMissingException: [.marvel-2014.01.29]
type[indices_stats] missing: trying to auto create
mapping, but dynamic mapping is disabled
at
org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:343)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:352)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
156)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Clearly that one's due to dynamic type mappings being disabled.
If you remove both of those configuration options, marvel appears to run
quite happily. 
Possible solutions:
- Remove the offending configuration options!
This isn't really an option for us, because we rely on these two
configuration settings in our production environment to help ensure
spurious indices aren't created accidentally and to ensure our type
mappings aren't accidentally corrupted due to extra json data being
submitted for indexing (also to help void incorrect type sniffing).
- marvel could explicitly create it's indices and update its type mappings
rather than relying on auto index creation and dynamic type mappings
To me, this feels like the "most correct" way to handle this. Clearly
marvel knows what index/indices it wants to have. And if it were to
explicitly check & create them (rather than relying on auto index
creation), it could also specify its type mapping instead of relying on
dynamic type mapping. A little tedious to define it explicitly, sure, but
then marvel would be compatible with a wider set of configurations
out-of-the-box.
-
marvel could check for these potentially conflicting settings and notify
you during plugin startup (or the ES the logs or the marvel plugin UI) of
the problem along with a workaround (use the auto_create_index whitelist,
etc.)? -
Other solutions I'm missing/not aware of?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f530a591-ec3c-4304-80c4-94960ff698eb%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
