Why Are Index Rollover Sequence Numbers Skipping in APM Data Stream?

Elastic Observability APM
,
1

Kibana version: 8.9.0
Elasticsearch version: 8.9.0
APM Server version: 8.9.0
APM Agent language and version: Node.js
Browser version: Chrome 137.0.0.0
Original install method: Deployed via ECK on AKS
Fresh install or upgraded from other version?: Fresh install

I'm using ECK operator for managing Elasticsearch, APM Server, and Kibana.
Self-managed hot/warm architecture.
ILM and data streams are enabled by default.

I’m observing that the backing indices of the APM data stream are not incrementing linearly.

Here is an example:

"
.ds-logs-apm.error-default-2025.06.20-000001
.ds-logs-apm.error-default-2025.06.21-000002
.ds-logs-apm.error-default-2025.06.22-000003
.ds-logs-apm.error-default-2025.06.23-000005
.ds-logs-apm.error-default-2025.06.24-000007
.ds-logs-apm.error-default-2025.06.25-000008
"

I expected these numbers to increment linearly. Is it normal for some numbers to be skipped?
If yes, under what conditions do those gaps occur?

2

Yes, this is normal behavior, the number in the end of the backing index name is the generation and in some cases it can skip some numbers as described in the documentation.

As mentioned in the docs:

The generation of the data stream can change without a new index being added to the data stream (e.g. when an existing backing index is shrunk). This means the backing indices for some generations will never exist. You should not derive any intelligence from the backing indices names.

It is not described exactly what actions can make a generation to be skipped, but this is normal and the user should not worry about it.

1 Like
3

Thank you for the clarification and for linking the documentation. :+1: