Upgrade elasticsearch from version 7.16 to 7.17

p_alavi
Wed, 29 Jan 2025 15:25:17 +0000
Kibana

Hi all,

I recently upgraded my Elasticsearch cluster from version 7.16 to 7.17 and encountered several issues. Kibana is not loading properly, and I am receiving license-related errors and missing authentication credentials. I manually closed some system indices like .kibana_task_manager and .kibana-event-log-* as a troubleshooting step. Here are some of the errors I’ve observed in the logs:

License errors preventing access to Kibana and certain API endpoints.

{
  "type": "log",
  "@timestamp": "2025-01-29T15:45:55+03:30",
  "tags": [
    "error",
    "plugins",
    "security",
    "authentication"
  ],
  "pid": 15802,
  "message": "License is not available, authentication is not possible."
}

Missing authentication credentials for some requests.

{
  "type": "log",
  "@timestamp": "2025-01-29T15:45:55+03:30",
  "tags": [
    "warning",
    "plugins",
    "securitySolution"
  ],
  "pid": 15802,
  "message": "Unable to verify endpoint policies in line with license change: failed to fetch package policies: missing authentication credentials for REST request [/.kibana_7.17.0_7.17.0/_search?from=0&rest_total_hits_as_int=true&size=100]: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/.kibana_7.17.0_7.17.0/_search?from=0&rest_total_hits_as_int=true&size=100]"
}

Random 503 errors and cluster health issues.

{
  "type": "response",
  "@timestamp": "2025-01-29T15:45:55+03:30",
  "tags": [],
  "pid": 15802,
  "method": "post",
  "statusCode": 503,
  "req": {
    "url": "/api/monitoring/v1/clusters/kVfChQJ8TWWjr5Mrbw_nEg",
    "method": "post",
    "headers": {
      "x-real-ip": "192.168.7.178",
      "x-forwarded-for": "192.168.7.178",
      "host": "logsummerize.sw.shatel.com",
      "connection": "upgrade",
      "content-length": "101",
      "sec-ch-ua-platform": "\"Windows\"",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
      "sec-ch-ua": "\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"",
      "content-type": "application/json",
      "kbn-version": "7.17.0",
      "sec-ch-ua-mobile": "?0",
      "accept": "*/*",
      "origin": "https://logsummerize.sw.shatel.com",
      "sec-fetch-site": "same-origin",
      "sec-fetch-mode": "cors",
      "sec-fetch-dest": "empty",
      "referer": "https://logsummerize.sw.shatel.com/app/monitoring",
      "accept-encoding": "gzip, deflate, br, zstd",
      "accept-language": "en-US,en;q=0.9,fa;q=0.8"
    },
    "remoteAddress": "172.22.129.189",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "referer": "https://logsummerize.sw.shatel.com/app/monitoring"
  },
  "res": {
    "statusCode": 503,
    "responseTime": 129,
    "contentLength": 86
  },
  "message": "POST /api/monitoring/v1/clusters/kVfChQJ8TWWjr5Mrbw_nEg 503 129ms - 86.0B"
}

Has anyone faced similar issues after upgrading or has any suggestions on how to resolve these errors, particularly related to the system indices and license validation?

I appreciate any help or insights!

14 posts - 4 participants