Marvel agent and index.mapper.dynamic: false

Elastic Stack Elasticsearch
1

Hello,
I'm trying to upgrade to Marvel 2.3.3 with Elasticsearch 2.3.3 but marvel-agent seems not able to create its indices.
These are the errors logs:

[2016-05-25 17:16:01,960][ERROR][marvel.agent             ] [APPS] background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
        at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
        at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
        at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution:
[0]: index [.marvel-es-1-2016.05.25], type [cluster_state], id [null], message [[.marvel-es-1-2016.05.25] IndexNotFoundException[no such index]]
[1]: index [.marvel-es-1-2016.05.25], type [node], id [null], message [[.marvel-es-1-2016.05.25] IndexNotFoundException[no such index]]
[2]: index [.marvel-es-data-1], type [node], id [NVxAuBdcSIG_UCGVti32Dg], message [[.marvel-es-data-1] IndexNotFoundException[no such index]]
[3]: index [.marvel-es-1-2016.05.25], type [indices_stats], id [null], message [[.marvel-es-1-2016.05.25] IndexNotFoundException[no such index]]

I have only 1 server in my cluster...and as you can see, the marvel indices are not created.

I also set a custom template for marvel indices:

curl -XGET http://localhost:9200/_template/custom_marvel?pretty
{
  "custom_marvel" : {
    "order" : 1,
    "template" : ".marvel*",
    "settings" : {
      "index" : {
        "number_of_replicas" : "0",
        "mapper" : {
          "dynamic" : "true"
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

but this doesn't fix the problem.

The only solution seems to comment the index.mapper.dynamic in the main elasticsearch configuration... and, in this situation, the marvel indices are created:

green  open   .marvel-2016.05.25        1   0      13815            0     20.1mb         20.1mb
green  open   .marvel-es-1-2016.05.25   1   0         29            3     87.8kb         87.8kb

...but I need this option.

Any idea how to fix the problem?

This is my elasticsearch.yml:

cluster.name: Test
node.name: APPS
path.data: /data/elasticsearch
bootstrap.mlockall: true
action.auto_create_index: .marvel-*, kibana-int
index.mapper.dynamic: false
index.number_of_shards: 1
index.number_of_replicas: 0
marvel.history.duration: 7

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:
  - "apps.test.dev"
2

Hi Francesco -

It looks like you created a custom template for Marvel but in our Configuring Marvel documentation, we state the following:

Only set the number_of_shards and number_of_replicas in the settings section. Overriding other Marvel template settings could cause your Marvel dashboards to stop working correctly.

With that said, could you share with us how you created the template (i.e. CURL command)?

Also, the following does not look right:

action.auto_create_index: .marvel-*, kibana-int

as you need to have '+' before the index names to allow those indices ('-' is to disallow). Please refer to Automatic Index Creation page for more information.

Thanks,
Bohyun

3

Hi Bohyun,
this configuration was from an elasticsearch 1.6/1.7 installation (where the marvel indices were correctly created) upgraded to 2.3.3 (where I have the problem).

Btw, I started from a clean situation in a vagrant machine with elasticsearch 2.3 and marvel-agent (ubuntu 14.04 and es deb package). With no options set in elasticsearch.yml, the marvel indices are created.
If I add these options:
action.auto_create_index: +.marvel-*, +kibana-int
index.mapper.dynamic: false
I have the same errors pasted in the first post.

I suppose the problem is the index.mapper.dynamic set to false.

This happens with or without the custom template for marvel indices (where I'm trying to disable the option for marvel indices, hope withour errors) that I can set with:

 curl -XPUT http://localhost:9200/_template/custom_marvel -d '{"template":".marvel*","order":1,"settings":{"number_of_replicas": 0,"mapper.dynamic":true}}'

Finally, my question :slight_smile:
can I use the marvel agent with the index.mapper.dynamic set to false in the main elasticsearch configuration (if this is the problem as it seems)?

Thanks,
Francesco

4

Hi Francesco -

The option index.mapper.dynamic is used to disable automatic type creation, and setting the index.mapper.dynamic to false will disable automatic mapping. It looks like to continue in using the template you have, it logically makes sense to set that to false.

I would still like to point out that it is not recommended to override the number_of_shards and number_of_replicas in Marvel templates.

Thanks,
Bohyun

1 Like
6

I didn't find anything about this on the documentation,
but on Elasticsearch 2.2, having marvel and index.mapper.dynamic: false was working fine.

After upgrading to 2.4, I had to remove this option, because Marvel was going nuts, only throwing IndexNotFoundException.

7

Marvel, particularly 2.x, benefits a good amount from dynamic mappings. We explicitly add [most of] what we use to the mappings, but we also send a bunch of data that we don't currently use.

I have not tried it, but you can probably get around this and still disable dynamic mappings globally by tweaking the Marvel templates' settings to explicitly include "index.mapper.dynamic": true.

It's something that we should probably add too. Thanks for calling it out.

2 Likes