Too many marvel indices with default per-day idx generation

Elastic Stack Elasticsearch
1

Greetings. There used to be a option for marvel in 1.x version that control the format Marvel uses to create its data index: marvel.agent.exporter.es.index.timeformat. I can therefore set that to YYYY.MM so that Marvel just create one index for the whole month data.
After I upgraded ELK to 2.x this option seems to be not effective any longer, as the new marvel begins to create index every day like YYYY.MM.DD. Is there any way to get back to my original index format setting?

2

Hi zkun,

The setting index.name.time_format can be configure at exporter level and it accepts a time format like YYYY, YYYY.MM, YYYY-MM-dd... default value is YYYY.MM.dd:

marvel.agent.exporters:
  id2:
    type: http
    host: [ "http://domain:port",... ]
    index.name.time_format: "YYYY.MM.dd"

We'll update the documentation to make this more clear.

1 Like
3

Thank you for the kind tip.
Then how should I config marvel app in kibana to get data from new indices instead of the old .marvel-es-YYYY.MM.dd? Now it's totally blank in marvel dashboard.

4

Update with my discovery yesterday:
There are several marvel index related init js script in kibana marvel plugin folder installedPlugins/marvel/. And I noticed that they still look for a pattern like YYYY.MM.dd although the format has been changed under marvel.agent.exporters in ELS config. But the dashboard keeps blank even after changing those in js to YYYY.MM
In the 1.x marvel the pattern could be set at a folded place in marvel dashboard. Hope the new marvel will inherit this by adding a brief config panel.

5

Hi zkun, we have an issue in Marvel 2.x where the UI doesn't follow the config setting for index.name.time_format, but we have identified a fix for it, and will have this fixed in a future version.

6

Thank you for the kind reply. Is this issue tracked on github so that I can check the progress of resolution?

7

We're planning to release a fix for this in the next Marvel version early next year.

I'm also curious, how much history do you have that the number of one-per-day indices are a problem?

8

We did use default index pattern settings at first. But as it turned out to be 100+ indices after 2 month running we switched to month-based idx pattern, given that there're only 2 data nodes to hold all indices.

9

I saw the fix has already been included in the latest release. Thanks!