Hi dears,
this our first post here.
We are quite new to Kibana and we hit a couple of problems:
We currently keep daily indexes in Kibana, monitoring log files with filebeat. Mapping is static.
Problems
- When we reindex old indexes to some new mapping, a large amount of docs is not copied to the new indexes.
- Somedays no new index is created even though logs are coming (a limit is reached? It is strange that we have run out of resources). We have to delete old indexes in order to be able to create new indexes.
// Remapping and reindexing
After adding a new field to the mapping, we'd like to reindex the old indexes, in order to extend the new mapping to old data.
We follow this elastic guide in order to bulk reindex every index: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#_reindex_daily_indices
The procedure works for a single index, meaning that every doc is copied to the new index.
When selecting multiple indexes (using as "source" an explicit array of selected indexes, or a wildcard), the indexes are indeed created, but only a small amount of docs are created in the new indexes.
The amount of docs actually created is not consistent across trials.
Some indexes have incompatible fields, due to some old mappings, and return errors. But this should not be a problem, since when reindexing the single problematic index all docs are copied except for the problematic ones, as expected.
Any hint?