Metricbeat Aggs fail after adding Index Routing

[ritchierich (Rich Horace)]

Earlier this week, I updated my v 5.3.2 cluster configuration for Hot/Warm Architecture. I added "index.routing.allocation.require.box_type": "hot" to all existing daily indices then updated my logstash and metricbeat templates.

First Day: metricbeat-mongo-2017.05.18 index created without any issues.
Second Day: metricbeat-mongo-2017.05.19 index created but...

• Aggs would return empty in both Kibana Visualization and directly querying Elasticsearch.
• All Data was visible in Kibana Discovery
• Tried refreshing index field list in Kibana Management

My solution:

1. Delete metricbeat template
2. Stopping all log indexing
3. Reindex existing metricbeat indices to New Indices
4. Delete Old Index
5. Start all log indexing
6. Aggregation worked

Meticbeat Template snippet if where I added Index Routing:

{
  "mappings": {
    "_default_": {
	 .... }
  },
 "order": 0,
  "settings": {
    "index.mapping.total_fields.limit": 10000,
    "index.refresh_interval": "5s",
    "index.routing.allocation.require.box_type": "hot"
  },
  "template": "metricbeat-*"
}

Am I missing something on adding the Index Routing?

Thanks in advance!

Cheers,
Rich

[ruflin]

Hi Rich

My first gut feeling is that there could be something wrong with the template. Are you sure the right template was loaded? Or were both templates loaded and now conflict with each others? Could you check how many templates you have in ES?

How do you load the template? Do you let Metricbeat doing it or do you load it manually?

[ritchierich (Rich Horace)]

@ruflin Thanks for the response. I haven't had the opportunity to test further. My workaround is to manually set routing after the daily index is created.

I'll do some process of elimination with the metricbeat template to see if I can figure it out.

Cheers,
Rich

[ruflin]

Thanks, keep us posted.

[ritchierich (Rich Horace)]

I haven't had the opportunity to test yet. My solution so far has been to manually set box_type for metricbeat indices.