Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0)

Elastic Observability APM
,
1

I have migrated data from an 8.2.0 cluster to a 8.7.1 cluster. Initially i had some problems with APM saying the ingestion pipelines did not exist.

{"log.level":"error","@timestamp":"2023-05-26T13:23:03.102Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (illegal_argument_exception): pipeline with id [traces-apm.rum-8.2.0] does not exist","service.name":"apm-server","ecs.version":"1.6.0"}

I had to manually create the ingestion pipelines 8.2.0 on the new cluster. But now im facing a new problem.

apiversion: apm.k8s.elastic.co/v1
kind: apmserver
metadata:
  name: apm-server-prod
  namespace: elastic-system
spec:
  version: 8.7.1
  count: 1
  elasticsearchref:
    name: "elasticsearch-prod"
  kibanaref:
    name: "kibana-prod"
  http:
    service:
      spec:
        type: nodeport
  config:
    http.enabled: true
    http.port: 5068
    monitoring.enabled: false
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
        anonymous:
          rate_limit:
            event_limit:  300
            ip_limit:  1000
          allow_service: ["admin-dashboard", "frontend"]
    apm-server.rum.enabled: true
    apm-server.rum.allow_origins: ["*"]

{"log.level":"error","@timestamp":"2023-05-26T17:17:47.853Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}
2

Hi @Kay_Khan,

Thank you for reaching out!

It seems there is an indexing error, you might need to perform a rollover:


POST metrics-apm.service_destination.10m-default/_rollover
POST metrics-apm.service_summary.10m-default/_rollover
POST metrics-apm.service_transaction.10m-default/_rollover
POST metrics-apm.transaction.10m-default/_rollover


POST metrics-apm.service_destination.60m-default/_rollover
POST metrics-apm.service_summary.60m-default/_rollover
POST metrics-apm.service_transaction.60m-default/_rollover
POST metrics-apm.transaction.60m-default/_rollover

Note: not all of the above data streams might exist

Let me know if the problem goes away after applying the rollovers

Thanks,
Alberto

3

I see the following data streams, do i rollover all of these?

image
image1668×878 101 KB

You can see the first error was to do with [traces-apm.rum-8.2.0

4

Hi we ran

POST metrics-apm.service_destination.1m-default/_rollover
POST metrics-apm.service_summary.1m-default/_rollover
POST metrics-apm.service_transaction.1m-default/_rollover
POST metrics-apm.transaction.1m-default/_rollover

But still getting the following error:

{"log.level":"error","@timestamp":"2023-05-30T07:18:04.731Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}
5

Hi @Kay_Khan,

It turns out there is a bug in the APM server that causes wrong ILM policy names to be generated, a bug which is fixed in 8.8.0.

The workaround to fix this in 8.7.x is to create the missing ILM policies, in Devtools, perform the following 12 HTTP requests:

PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.1m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.10m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "14d"
          }
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.60m
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "390d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Let me know if this helps you

Cheers,
Alberto

6

Hi Alberto_Delgado thank you for the response

We just applied those 12 http requests

# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_destination_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_transaction_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.transaction_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.1m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.10m 200 OK
{
  "acknowledged": true
}
# PUT _ilm/policy/metrics-apm.service_summary_interval_metrics-default_policy.60m 200 OK
{
  "acknowledged": true
}

We rebooted our apm server and it is still shwoing the following errors failing to ingest data becuase of version mismatch.

{"log.level":"error","@timestamp":"2023-05-30T08:52:46.107Z","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (fail_processor_exception): Document produced by APM Server v8.7.1, which is newer than the installed APM integration (v8.2.0). The APM integration must be upgraded.","service.name":"apm-server","ecs.version":"1.6.0"}

image
image945×363 50 KB

image
image1570×942 51.8 KB

Although we are not sure why its suggested we have 8.2.0 installed when we can see things showing 8.7.1 is installed

7

Okay so we applied a /rollover of any/all apm datastreams.

We saw some of the default_pipeline for those index were showing a value of 8.2.0. So maybe apm thought that version 8.2.0 of the integration was installed because of the value of default_pipeline.

After applying the rollover we saw the value switch to 8.7.1 of the new indexse in the data stream. Then rebooting apm, we no longer see the version mismatch error and appears to be working correctly.

1 Like
9

We're using APM server with custom namespace. Noticed the same - indices were using outdated ingest pipelines - however i had to manually change datastream template to use new pipeline version and perform a rollover. Then it started working. This post pushed me in the right direction!

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.