# Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0)

**URL:** <https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568>\
**Category:** APM\
**Tags:** rum, server\
**Created:** [May 29, 2023, 9:28am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568 "2023-05-29T09:28:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [May 29, 2023, 9:28am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/1 "2023-05-29T09:28:09Z")

</div>

I have migrated data from an 8.2.0 cluster to a 8.7.1 cluster. Initially i had some problems with APM saying the ingestion pipelines did not exist.

```auto
{"log.level":"error","@timestamp":"2023-05-26T13:23:03.102Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (illegal_argument_exception): pipeline with id [traces-apm.rum-8.2.0] does not exist","service.name":"apm-server","ecs.version":"1.6.0"}

```

I had to manually create the ingestion pipelines 8.2.0 on the new cluster. But now im facing a new problem.

```auto
apiversion: apm.k8s.elastic.co/v1
kind: apmserver
metadata:
  name: apm-server-prod
  namespace: elastic-system
spec:
  version: 8.7.1
  count: 1
  elasticsearchref:
    name: "elasticsearch-prod"
  kibanaref:
    name: "kibana-prod"
  http:
    service:
      spec:
        type: nodeport
  config:
    http.enabled: true
    http.port: 5068
    monitoring.enabled: false
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
        anonymous:
          rate_limit:
            event_limit: 300
            ip_limit: 1000
          allow_service: ["admin-dashboard", "frontend"]
    apm-server.rum.enabled: true
    apm-server.rum.allow_origins: ["*"]

```

```auto
{"log.level":"error","@timestamp":"2023-05-26T17:17:47.853Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![Alberto\_Delgado](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alberto_delgado/32/98488_2.png) [@Alberto\_Delgado](https://discuss.elastic.co/u/Alberto_Delgado)\
**Post date:** [May 29, 2023, 5:04pm UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/2 "2023-05-29T17:04:29Z")

</div>

Hi @Kay_Khan,

Thank you for reaching out!

It seems there is an indexing error, you might need to perform a rollover:

```auto

POST metrics-apm.service_destination.10m-default/_rollover
POST metrics-apm.service_summary.10m-default/_rollover
POST metrics-apm.service_transaction.10m-default/_rollover
POST metrics-apm.transaction.10m-default/_rollover

```

```auto

POST metrics-apm.service_destination.60m-default/_rollover
POST metrics-apm.service_summary.60m-default/_rollover
POST metrics-apm.service_transaction.60m-default/_rollover
POST metrics-apm.transaction.60m-default/_rollover

```

**Note** : not all of the above data streams might exist

Let me know if the problem goes away after applying the rollovers

Thanks,  
Alberto

---

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [May 29, 2023, 7:12pm UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/3 "2023-05-29T19:12:19Z")

</div>

I see the following data streams, do i rollover all of these?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/7/47d08f748f428353896bdf4dfaeeb9d92c04fa0c.png)

You can see the first error was to do with `[traces-apm.rum-8.2.0`

---

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [May 30, 2023, 7:19am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/4 "2023-05-30T07:19:16Z")

</div>

Hi we ran

```auto
POST metrics-apm.service_destination.1m-default/_rollover
POST metrics-apm.service_summary.1m-default/_rollover
POST metrics-apm.service_transaction.1m-default/_rollover
POST metrics-apm.transaction.1m-default/_rollover

```

But still getting the following error:

```auto
{"log.level":"error","@timestamp":"2023-05-30T07:18:04.731Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![Alberto\_Delgado](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alberto_delgado/32/98488_2.png) [@Alberto\_Delgado](https://discuss.elastic.co/u/Alberto_Delgado)\
**Post date:** [May 30, 2023, 8:32am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/5 "2023-05-30T08:32:58Z")

</div>

Hi @Kay_Khan,

It turns out there is a bug in the APM server that causes wrong ILM policy names to be generated, a bug which is fixed in 8.8.0.

The workaround to fix this in 8.7.x is to create the missing ILM policies, in Devtools, perform the following 12 HTTP requests:

```auto
PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

```

Let me know if this helps you

Cheers,  
Alberto

---

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [May 30, 2023, 8:54am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/6 "2023-05-30T08:54:23Z")

</div>

> [@Alberto\_Delgado](#):
>
> `e`

Hi [Alberto\_Delgado](https://discuss.elastic.co/u/Alberto_Delgado) thank you for the response

We just applied those 12 http requests

```auto
# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}

```

We rebooted our apm server and it is still shwoing the following errors failing to ingest data becuase of version mismatch.

```auto
{"log.level":"error","@timestamp":"2023-05-30T08:52:46.107Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4ba7e843f88e9ce1df30b9f6c953a54adfdb126a.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/1573aa21ae38ba1c99ec76af83bdd8995c9df601.png)

Although we are not sure why its suggested we have 8.2.0 installed when we can see things showing 8.7.1 is installed

---

<div class="post-metadata">

**Author:** ![Kay\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_khan/32/45028_2.png) [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\
**Post date:** [May 30, 2023, 9:23am UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/7 "2023-05-30T09:23:29Z")

</div>

Okay so we applied a /rollover of any/all _apm_ datastreams.

We saw some of the default\_pipeline for those index were showing a value of 8.2.0. So maybe apm thought that version 8.2.0 of the integration was installed because of the value of default\_pipeline.

After applying the rollover we saw the value switch to 8.7.1 of the new indexse in the data stream. Then rebooting apm, we no longer see the version mismatch error and appears to be working correctly.

---

<div class="post-metadata">

**Author:** ![Marko\_Todoric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marko_todoric/32/52975_2.png) [@Marko\_Todoric](https://discuss.elastic.co/u/Marko_Todoric)\
**Post date:** [June 14, 2023, 9:36pm UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/9 "2023-06-14T21:36:41Z")

</div>

We're using APM server with custom namespace. Noticed the same - indices were using outdated ingest pipelines - however i had to manually change datastream template to use new pipeline version and perform a rollover. Then it started working. This post pushed me in the right direction!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2023, 9:37pm UTC](https://discuss.elastic.co/t/document-produced-by-apm-server-v8-7-1-which-is-newer-than-the-installed-apm-integration-v8-2-0/334568/10 "2023-07-12T21:37:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.