Marvel plugin incompatible with action.auto_create_index: false and index.mapper.dynamic: false configuration settings


(Dane Bertram) #1

Environment: I'm running ES 0.90.10 on a Windows 7 Pro box (my dev machine).

Steps:

  1. Add the following to your [es directory]/config/elasticsearch.yml
    before installing marvel:

action.auto_create_index: false
index.mapper.dynamic: false

  1. Install marvel plugin
  2. Restart your ES cluster
  3. Navigate to http://localhost:9200/_plugin/marvel

Expected:

Either a working marvel dashboard or a very explicit warning/error message
that marvel failed to create its index.

Bonus: detect the settings above and give specific remedies in the
notification (remove those settings, use the auto_create_index whitelist to
allow marvel to create it's own indices, etc.).

Actual:

  1. Empty marvel dashboard (page renders just fine, just no data to
    display), with a short-lived message at the top of the dashboard: "No
    results There were no results because no indices were found that match your
    selected time span"
  2. The following error repeating in your ES logs 4 times every second:

[2014-01-29 10:21:11,639][ERROR][marvel.agent.exporter ] error sending
data
java.io.FileNotFoundException: http://localhost:9200/_bulk
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.sendCloseExportingConnection(ESExporter.java:232)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportClusterStats(ESExporter.java:172)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportClusterStats(AgentService.java:214)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:180)
at java.lang.Thread.run(Thread.java:744)

This error is a symptom of marvel not being able to create it's own index
due to action.auto_create_index: false in our elasticsearch.yml

My next step was to turn off the action.auto_create_index: false setting in
elasticsearch.yml and re-start the cluster.

This resulted in marvel being able to create it's index, but still failing
to index any data.

  1. Same issue with the marvel dashboard (no data, no warnings/errors)
  2. Different error message in the ES logs 4 times/second:

[2014-01-29 10:24:54,793][DEBUG][action.bulk ] [Princess
Python] [.marvel-2014.01.29][0] failed to execute
bulk item (index) index
{[.marvel-2014.01.29][indices_stats][5eVda-3SRBK2TKmjHzFpBA],
source[{"@timestamp":"2014-01-29T1
5:24:54.786Z","cluster_name":"elasticsearch","_source_node":{"id":"LBDVUL_eRt27pnVCno7NBQ","name":"Princess
Python","tra
nsport_address":"inet[dane.hq.fogcreek.com/10.1.1.167:9300]","ip":"10.1.1.167","host":"dane.hq.fogcreek.com","ip_port":"
10.1.1.167:9300"},"primaries":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_m
illis":2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_mil
lis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total
":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"qu
ery_current":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"cur
rent_size_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"to
tal":19,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"tota
l_time_in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"
fielddata":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory
in_bytes":0}},"total":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_millis":
2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_millis":0,
"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"mi
ssing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"query_cur
rent":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_si
ze_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"total":19
,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"total_time

in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fieldda
ta":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory_in_byt
es":0}}}]}
org.elasticsearch.indices.TypeMissingException: [.marvel-2014.01.29]
type[indices_stats] missing: trying to auto create
mapping, but dynamic mapping is disabled
at
org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:343)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:352)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)

    at 

org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
156)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Clearly that one's due to dynamic type mappings being disabled.

If you remove both of those configuration options, marvel appears to run
quite happily. :slight_smile:

Possible solutions:

  1. Remove the offending configuration options! :stuck_out_tongue:

This isn't really an option for us, because we rely on these two
configuration settings in our production environment to help ensure
spurious indices aren't created accidentally and to ensure our type
mappings aren't accidentally corrupted due to extra json data being
submitted for indexing (also to help void incorrect type sniffing).

  1. marvel could explicitly create it's indices and update its type mappings
    rather than relying on auto index creation and dynamic type mappings

To me, this feels like the "most correct" way to handle this. Clearly
marvel knows what index/indices it wants to have. And if it were to
explicitly check & create them (rather than relying on auto index
creation), it could also specify its type mapping instead of relying on
dynamic type mapping. A little tedious to define it explicitly, sure, but
then marvel would be compatible with a wider set of configurations
out-of-the-box.

  1. marvel could check for these potentially conflicting settings and notify
    you during plugin startup (or the ES the logs or the marvel plugin UI) of
    the problem along with a workaround (use the auto_create_index whitelist,
    etc.)?

  2. Other solutions I'm missing/not aware of?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f530a591-ec3c-4304-80c4-94960ff698eb%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Dane Bertram) #2

Ah, I see the index auto creation setting is mentioned in the
docs: http://www.elasticsearch.org/guide/en/marvel/current/#_other_relevant_elasticsearch_settings

The dynamic type mapping setting should probably be added there as well,
but I still feel like the plugin should be more explicit/helpful when it
runs into this type of problem.

On Wednesday, 29 January 2014 10:41:16 UTC-5, Dane Bertram wrote:

Environment: I'm running ES 0.90.10 on a Windows 7 Pro box (my dev
machine).

Steps:

  1. Add the following to your [es directory]/config/elasticsearch.yml
    before installing marvel:

action.auto_create_index: false
index.mapper.dynamic: false

  1. Install marvel plugin
  2. Restart your ES cluster
  3. Navigate to http://localhost:9200/_plugin/marvel

Expected:

Either a working marvel dashboard or a very explicit warning/error message
that marvel failed to create its index.

Bonus: detect the settings above and give specific remedies in the
notification (remove those settings, use the auto_create_index whitelist to
allow marvel to create it's own indices, etc.).

Actual:

  1. Empty marvel dashboard (page renders just fine, just no data to
    display), with a short-lived message at the top of the dashboard: "No
    results There were no results because no indices were found that match your
    selected time span"
  2. The following error repeating in your ES logs 4 times every second:

[2014-01-29 10:21:11,639][ERROR][marvel.agent.exporter ] error sending
data
java.io.FileNotFoundException: http://localhost:9200/_bulk
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.sendCloseExportingConnection(ESExporter.java:232)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.exportClusterStats(ESExporter.java:172)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportClusterStats(AgentService.java:214)
at
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:180)
at java.lang.Thread.run(Thread.java:744)

This error is a symptom of marvel not being able to create it's own index
due to action.auto_create_index: false in our elasticsearch.yml

My next step was to turn off the action.auto_create_index: false setting
in elasticsearch.yml and re-start the cluster.

This resulted in marvel being able to create it's index, but still failing
to index any data.

  1. Same issue with the marvel dashboard (no data, no warnings/errors)
  2. Different error message in the ES logs 4 times/second:

[2014-01-29 10:24:54,793][DEBUG][action.bulk ] [Princess
Python] [.marvel-2014.01.29][0] failed to execute
bulk item (index) index
{[.marvel-2014.01.29][indices_stats][5eVda-3SRBK2TKmjHzFpBA],
source[{"@timestamp":"2014-01-29T1
5:24:54.786Z","cluster_name":"elasticsearch","_source_node":{"id":"LBDVUL_eRt27pnVCno7NBQ","name":"Princess
Python","tra
nsport_address":"inet[dane.hq.fogcreek.com/10.1.1.167:9300
]","ip":"10.1.1.167","host":"dane.hq.fogcreek.com","ip_port":"
10.1.1.167:9300
"},"primaries":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_m

illis":2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_mil

lis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total

":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"qu

ery_current":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"cur

rent_size_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"to

tal":19,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"tota

l_time_in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"

fielddata":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory

_in_bytes":0}},"total":{"docs":{"count":15249,"deleted":720},"store":{"size_in_bytes":5790385,"throttle_time_in_millis":

2},"indexing":{"index_total":37,"index_time_in_millis":430,"index_current":0,"delete_total":0,"delete_time_in_millis":0,

"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"mi

ssing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":120,"query_time_in_millis":353,"query_cur

rent":0,"fetch_total":96,"fetch_time_in_millis":20,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_si

ze_in_bytes":0,"total":1,"total_time_in_millis":262,"total_docs":27,"total_size_in_bytes":1007730},"refresh":{"total":19

,"total_time_in_millis":1049},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":20,"total_time_

in_millis":26},"filter_cache":{"memory_size_in_bytes":2200,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fieldda

ta":{"memory_size_in_bytes":174036,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":38,"memory_in_byt
es":0}}}]}
org.elasticsearch.indices.TypeMissingException: [.marvel-2014.01.29]
type[indices_stats] missing: trying to auto create
mapping, but dynamic mapping is disabled
at
org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:343)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:352)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)

    at 

org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:
156)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationActi
on$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Clearly that one's due to dynamic type mappings being disabled.

If you remove both of those configuration options, marvel appears to run
quite happily. :slight_smile:

Possible solutions:

  1. Remove the offending configuration options! :stuck_out_tongue:

This isn't really an option for us, because we rely on these two
configuration settings in our production environment to help ensure
spurious indices aren't created accidentally and to ensure our type
mappings aren't accidentally corrupted due to extra json data being
submitted for indexing (also to help void incorrect type sniffing).

  1. marvel could explicitly create it's indices and update its type
    mappings rather than relying on auto index creation and dynamic type
    mappings

To me, this feels like the "most correct" way to handle this. Clearly
marvel knows what index/indices it wants to have. And if it were to
explicitly check & create them (rather than relying on auto index
creation), it could also specify its type mapping instead of relying on
dynamic type mapping. A little tedious to define it explicitly, sure, but
then marvel would be compatible with a wider set of configurations
out-of-the-box.

  1. marvel could check for these potentially conflicting settings and
    notify you during plugin startup (or the ES the logs or the marvel plugin
    UI) of the problem along with a workaround (use the auto_create_index
    whitelist, etc.)?

  2. Other solutions I'm missing/not aware of?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c520e6e8-1750-4c64-880c-3c330f8f8771%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Dane Bertram) #3

Ah, I set the index auto create setting is mentioned in the docs for
marvel:
http://www.elasticsearch.org/guide/en/marvel/current/#_other_relevant_elasticsearch_settings

The dynamic type mapping setting should also probably be mentioned there,
but I still feel like the plugin could/should be more helpful when it runs
into this type of problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e2e2c06-34d0-429a-924a-5a8678021dab%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4