Kibana stuck in a crash loop

Glorified_SysAdmin · August 13, 2024, 1:58pm

Hello,

I have a EFK stack deployed in EKS via helm. I have 6 nodes assigned to my ES cluster - 3 master, 2 ingest and 1 primary. All of my ES nodes are up and running. However, my Kibana pod is stuck on restart loop, keeps starting and crashing. I checked out the log for my Kibana pod and below is the error message:

  Root causes:                                                                                                                                     │
│         parse_exception: No processor type exists with name [inference]                                                                              │
│     at KibanaTransport.request (/opt/bitnami/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)                                         │
│     at processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                            │
│     at KibanaTransport.request (/opt/bitnami/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)       │
│     at Ingest.putPipeline (/opt/bitnami/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ingest.js:139:16)                                     │
│     at ObservabilityAIAssistantService.<anonymous> (/opt/bitnami/kibana/node_modules/@kbn/observability-ai-assistant-plugin/server/service/index.js: type or paste code here

Seems like it is missing some indexes. How can I load back the missing indices? Thanks in advance!

yago82 · August 20, 2024, 11:39am

Hi,

Ensure that your ingest nodes have the ml role. This role is required to use Machine Learning features, including the inference processor.

Regards

Glorified_SysAdmin · August 20, 2024, 1:24pm

How would I add it? Just add the below to my ES yaml?

ingest:
    replicaCount: 2
    node.roles: [ ingest, ml ]

yago82 · August 20, 2024, 1:37pm

yes

Glorified_SysAdmin · August 20, 2024, 2:24pm

I added the above config to add ML roles to my ingest nodes. However, when I run GET /_cat/nodes I don't see the ml roles associated with my ingest nodes

d           - elasticsearch-data-0
cdfhilmrstw - elasticsearch-master-1
i           - elasticsearch-ingest-1
cdfhilmrstw * - elasticsearch-master-0
cdfhilmrstw - elasticsearch-master-2
i           - elasticsearch-ingest-0

yago82 · August 20, 2024, 2:25pm

l (machine learning node)

did you reboot the nodes?

Glorified_SysAdmin · August 20, 2024, 2:27pm

Doesn't I mean ingest?

Glorified_SysAdmin · August 20, 2024, 2:31pm

Yes, I re-installed the helm chart after making the config changes.

yago82 · August 20, 2024, 2:43pm

i (ingest node), l (machine learning node)

Glorified_SysAdmin · August 20, 2024, 3:06pm

gotcha. I don't see the config being applied. Seems as if master nodes hold all the roles. Ingest only ingest roles and data only data roles. I am not able to apply ml roles to ingest nodes.

Glorified_SysAdmin · August 20, 2024, 3:07pm

below is my ES yaml

elasticsearch:
  enabled: true
  master:
    replicaCount: 3
    heapSize: 16384m
    roles: ["master"]
    affinity: 
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: es-nodes
              operator: In
              values:
              - master
    tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "es-master"
      effect: "NoSchedule"
  data:
    replicaCount: 1
    heapSize: 16384m
    roles: ["data"]
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: es-nodes
              operator: In
              values:
              - data
    tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "es-data"
      effect: "NoSchedule"
  ingest:
    replicaCount: 2
    heapSize: 16384m
    roles: ["ingest", "ml"]
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: es-nodes
              operator: In
              values:
              - ingest
    tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "es-ingest"
      effect: "NoSchedule"

leandrojmp · August 20, 2024, 3:11pm

Do you have a paid license?

Machine Learning nodes requires a paid license.

Not sure this is the reason of your issue.

Can you share the complete Kibana log? Also, which version are you using?

Glorified_SysAdmin · August 20, 2024, 3:23pm

We have a basic license. Both ES and KIbana are running v8.10.3. Below is the kibana log:

e[38;5;6mkibana e[38;5;5m15:12:27.80 e[0m
e[38;5;6mkibana e[38;5;5m15:12:27.80 e[0me[1mWelcome to the Bitnami kibana containere[0m
e[38;5;6mkibana e[38;5;5m15:12:27.80 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mkibana e[38;5;5m15:12:27.81 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mkibana e[38;5;5m15:12:27.81 e[0m
e[38;5;6mkibana e[38;5;5m15:12:27.81 e[0me[38;5;2mINFO e[0m ==> ** Starting Kibana setup **
e[38;5;6mkibana e[38;5;5m15:12:27.83 e[0me[38;5;2mINFO e[0m ==> Configuring/Initializing Kibana...
e[38;5;6mkibana e[38;5;5m15:12:27.84 e[0me[38;5;2mINFO e[0m ==> Found mounted configuration directory
e[38;5;6mkibana e[38;5;5m15:12:27.85 e[0me[38;5;2mINFO e[0m ==> ** Kibana setup finished! **

e[38;5;6mkibana e[38;5;5m15:12:27.87 e[0me[38;5;2mINFO e[0m ==> ** Starting Kibana **
Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.10/production.html#openssl-legacy-provider
{"log.level":"info","@timestamp":"2024-08-20T15:12:28.879Z","log":{"logger":"elastic-apm-node"},"agentVersion":"3.49.1","env":{"pid":1,"proctitle":"/opt/bitnami/kibana/bin/../node/bin/node","os":"linux 5.10.215-203.850.amzn2.x86_64","arch":"x64","host":"ai-search-kibana-6dc5bbf494-dgf6p","timezone":"UTC+00","runtime":"Node.js v18.17.1"},"config":{"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.10.3","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"active":{"source":"start","value":true},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"logUncaughtExceptions":{"source":"start","value":true},"globalLabels":{"source":"start","value":[["kibana_uuid","af096441-1efd-49e4-90a1-62f7ccf865fa"],["git_rev","5aee3c4fba328838fcf0be6a3ff2248a4c0120dd"]],"sourceValue":{"kibana_uuid":"af096441-1efd-49e4-90a1-62f7ccf865fa","git_rev":"5aee3c4fba328838fcf0be6a3ff2248a4c0120dd"}},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"breakdownMetrics":{"source":"start","value":false},"captureSpanStackTraces":{"source":"start","sourceValue":false},"centralConfig":{"source":"start","value":false},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"propagateTracestate":{"source":"start","value":true},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v3.49.1"}
[2024-08-20T15:12:30.390+00:00][INFO ][root] Kibana is starting
[2024-08-20T15:12:30.555+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2024-08-20T15:12:43.786+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2024-08-20T15:12:43.794+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2024-08-20T15:12:43.795+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2024-08-20T15:12:43.795+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2024-08-20T15:12:43.989+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2024-08-20T15:12:44.047+00:00][INFO ][plugins-service] Plugin "securitySolutionServerless" is disabled.
[2024-08-20T15:12:44.047+00:00][INFO ][plugins-service] Plugin "serverless" is disabled.
[2024-08-20T15:12:44.048+00:00][INFO ][plugins-service] Plugin "serverlessObservability" is disabled.
[2024-08-20T15:12:44.048+00:00][INFO ][plugins-service] Plugin "serverlessSearch" is disabled.
[2024-08-20T15:12:44.478+00:00][INFO ][http.server.Preboot] http server running at http://:::5601
[2024-08-20T15:12:44.671+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2024-08-20T15:12:44.708+00:00][WARN ][config.deprecation] You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning.
[2024-08-20T15:12:44.709+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2024-08-20T15:12:46.826+00:00][INFO ][plugins-system.standard] Setting up [141] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,charts,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,watcher,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,lens,maps,discover,reporting,canvas,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverLogExplorer,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,observabilityAIAssistant,logsShared,observability,observabilityOnboarding,aiops,ml,uptime,synthetics,securitySolution,securitySolutionEss,infra,upgradeAssistant,monitoring,logstash,apm,enterpriseSearch,dashboardEnhanced,dataViewManagement,assetManager]
[2024-08-20T15:12:46.838+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: af096441-1efd-49e4-90a1-62f7ccf865fa
[2024-08-20T15:12:46.875+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2024-08-20T15:12:47.008+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.008+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2024-08-20T15:12:47.049+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.049+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2024-08-20T15:12:47.092+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.099+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.108+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2024-08-20T15:12:47.239+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.539+00:00][INFO ][plugins.alerting] Registering resources for context "stack".
[2024-08-20T15:12:47.587+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2024-08-20T15:12:47.590+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2024-08-20T15:12:47.675+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2024-08-20T15:12:47.730+00:00][INFO ][plugins.alerting] Registering resources for context "observability.slo".
[2024-08-20T15:12:47.779+00:00][INFO ][plugins.alerting] Registering resources for context "observability.uptime".
[2024-08-20T15:12:47.842+00:00][INFO ][plugins.alerting] Registering resources for context "security".
[2024-08-20T15:12:47.879+00:00][INFO ][plugins.securitySolution] {"id":"securitySolutionCases","name":"Cases","order":1100,"category":{"id":"securitySolution","label":"Security","order":4000,"euiIconType":"logoSecurity"},"app":["securitySolutionCases","kibana"],"catalogue":["securitySolution"],"cases":["securitySolution"],"privileges":{"all":{"api":["casesSuggestUserProfiles","bulkGetUserProfiles","securitySolutionFilesCasesCreate","securitySolutionFilesCasesRead","casesGetConnectorsConfigure"],"app":["securitySolutionCases","kibana"],"catalogue":["securitySolution"],"cases":{"create":["securitySolution"],"read":["securitySolution"],"update":["securitySolution"],"push":["securitySolution"]},"savedObject":{"all":["file","fileShare"],"read":["file","fileShare"]},"ui":["create_cases","read_cases","update_cases","push_cases","cases_connectors"]},"read":{"api":["casesSuggestUserProfiles","bulkGetUserProfiles","securitySolutionFilesCasesRead","casesGetConnectorsConfigure"],"app":["securitySolutionCases","kibana"],"catalogue":["securitySolution"],"cases":{"read":["securitySolution"]},"savedObject":{"all":[],"read":["file","fileShare"]},"ui":["read_cases","cases_connectors"]}},"subFeatures":[{"name":"Delete","privilegeGroups":[{"groupType":"independent","privileges":[{"api":["securitySolutionFilesCasesDelete"],"id":"cases_delete","name":"Delete cases and comments","includeIn":"all","savedObject":{"all":["file","fileShare"],"read":["file","fileShare"]},"cases":{"delete":["securitySolution"]},"ui":["delete_cases"]}]}]}]}
[2024-08-20T15:12:47.881+00:00][INFO ][plugins.securitySolution] {"id":"securitySolutionAssistant","name":"Elastic AI Assistant","order":1100,"category":{"id":"securitySolution","label":"Security","order":4000,"euiIconType":"logoSecurity"},"app":["securitySolutionAssistant","kibana"],"catalogue":["securitySolution"],"minimumLicense":"enterprise","privileges":{"all":{"api":[],"app":["securitySolutionAssistant","kibana"],"catalogue":["securitySolution"],"savedObject":{"all":[],"read":[]},"ui":["ai-assistant"]},"read":{"disabled":true,"savedObject":{"all":[],"read":[]},"ui":[]}},"subFeatures":[]}
[2024-08-20T15:12:47.898+00:00][INFO ][plugins.alerting] Registering resources for context "observability.logs".
[2024-08-20T15:12:47.900+00:00][INFO ][plugins.alerting] Registering resources for context "observability.metrics".
[2024-08-20T15:12:47.943+00:00][INFO ][plugins.alerting] Registering resources for context "observability.apm".
[2024-08-20T15:12:48.019+00:00][INFO ][plugins.assetManager] Asset manager plugin [tech preview] is NOT enabled
[2024-08-20T15:12:48.217+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Debian 11.8 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
[2024-08-20T15:12:48.327+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2024-08-20T15:12:48.328+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2024-08-20T15:12:48.380+00:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 31ms.
[2024-08-20T15:12:48.383+00:00][INFO ][savedobjects-service] [.kibana_analytics] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 28ms.
[2024-08-20T15:12:48.385+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 22ms.
[2024-08-20T15:12:48.387+00:00][INFO ][savedobjects-service] [.kibana_security_solution] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 27ms.
[2024-08-20T15:12:48.389+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 37ms.
[2024-08-20T15:12:48.391+00:00][INFO ][savedobjects-service] [.kibana_analytics] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 8ms.
[2024-08-20T15:12:48.392+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 1ms.
[2024-08-20T15:12:48.394+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 9ms.
[2024-08-20T15:12:48.394+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 0ms.
[2024-08-20T15:12:48.396+00:00][INFO ][savedobjects-service] [.kibana_security_solution] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 9ms.
[2024-08-20T15:12:48.396+00:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 0ms.
[2024-08-20T15:12:48.397+00:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 17ms.
[2024-08-20T15:12:48.398+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 1ms.
[2024-08-20T15:12:48.400+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 8ms.
[2024-08-20T15:12:48.402+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 8ms.
[2024-08-20T15:12:48.404+00:00][INFO ][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 15ms.
[2024-08-20T15:12:48.405+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 1ms.
[2024-08-20T15:12:48.406+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 10ms.
[2024-08-20T15:12:48.409+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 11ms.
[2024-08-20T15:12:48.411+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 6ms.
[2024-08-20T15:12:48.413+00:00][INFO ][savedobjects-service] [.kibana_ingest] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 46ms.
[2024-08-20T15:12:48.415+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 15ms.
[2024-08-20T15:12:48.416+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 14ms.
[2024-08-20T15:12:48.418+00:00][INFO ][savedobjects-service] [.kibana_ingest] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 5ms.
[2024-08-20T15:12:48.418+00:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 0ms.
[2024-08-20T15:12:48.420+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 5ms.
[2024-08-20T15:12:48.421+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2024-08-20T15:12:48.421+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 0ms.
[2024-08-20T15:12:48.421+00:00][INFO ][savedobjects-service] [.kibana_analytics] Migration completed after 66ms
[2024-08-20T15:12:48.422+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 6ms.
[2024-08-20T15:12:48.423+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2024-08-20T15:12:48.423+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 0ms.
[2024-08-20T15:12:48.424+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] Migration completed after 60ms
[2024-08-20T15:12:48.425+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 19ms.
[2024-08-20T15:12:48.426+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 17ms.
[2024-08-20T15:12:48.427+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 9ms.
[2024-08-20T15:12:48.430+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 19ms.
[2024-08-20T15:12:48.432+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 7ms.
[2024-08-20T15:12:48.432+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 0ms.
[2024-08-20T15:12:48.433+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2024-08-20T15:12:48.433+00:00][INFO ][savedobjects-service] [.kibana_security_solution] Migration completed after 73ms
[2024-08-20T15:12:48.434+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 8ms.
[2024-08-20T15:12:48.434+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 0ms.
[2024-08-20T15:12:48.435+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2024-08-20T15:12:48.435+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 86ms
[2024-08-20T15:12:48.442+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 12ms.
[2024-08-20T15:12:48.443+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2024-08-20T15:12:48.443+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 0ms.
[2024-08-20T15:12:48.443+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 91ms
[2024-08-20T15:12:48.445+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 18ms.
[2024-08-20T15:12:48.448+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 3ms.
[2024-08-20T15:12:48.448+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 0ms.
[2024-08-20T15:12:48.449+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2024-08-20T15:12:48.449+00:00][INFO ][savedobjects-service] [.kibana_ingest] Migration completed after 82ms
[2024-08-20T15:12:48.455+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2024-08-20T15:12:48.456+00:00][INFO ][plugins-system.standard] Starting [141] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,charts,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,watcher,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,lens,maps,discover,reporting,canvas,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,discoverLogExplorer,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,observabilityAIAssistant,logsShared,observability,observabilityOnboarding,aiops,ml,uptime,synthetics,securitySolution,securitySolutionEss,infra,upgradeAssistant,monitoring,logstash,apm,enterpriseSearch,dashboardEnhanced,dataViewManagement,assetManager]
[2024-08-20T15:12:49.048+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.1 scheduled with interval 1h
[2024-08-20T15:12:49.048+00:00][INFO ][plugins.fleet.fleet:check-deleted-files-task:1.0.1] Started with interval of [1d] and timeout of [2m]
[2024-08-20T15:12:49.078+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2024-08-20T15:12:49.142+00:00][INFO ][plugins.alerting] Installing ILM policy .alerts-ilm-policy
[2024-08-20T15:12:49.144+00:00][INFO ][plugins.alerting] Installing component template .alerts-framework-mappings
[2024-08-20T15:12:49.146+00:00][INFO ][plugins.alerting] Installing component template .alerts-legacy-alert-mappings
[2024-08-20T15:12:49.174+00:00][INFO ][plugins.alerting] Installing component template .alerts-ecs-mappings
[2024-08-20T15:12:49.186+00:00][INFO ][plugins.ruleRegistry] Installing component template .alerts-technical-mappings
[2024-08-20T15:12:51.604+00:00][INFO ][http.server.Kibana] http server running at http://:::5601
[2024-08-20T15:12:51.631+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
[2024-08-20T15:12:51.664+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2024-08-20T15:12:51.680+00:00][INFO ][plugins.telemetry] Telemetry collection is enabled. For more information on telemetry settings, refer to https://www.elastic.co/guide/en/kibana/8.10/telemetry-settings-kbn.html.
[2024-08-20T15:12:51.718+00:00][INFO ][plugins.ruleRegistry] Installing ILM policy .preview.alerts-security.alerts-policy
[2024-08-20T15:12:51.765+00:00][INFO ][status] Kibana is now degraded
[2024-08-20T15:12:51.776+00:00][INFO ][plugins.ruleRegistry] Installing component template .preview.alerts-security.alerts-mappings
[2024-08-20T15:12:51.789+00:00][INFO ][plugins.alerting] Installing component template .alerts-stack.alerts-mappings
[2024-08-20T15:12:51.790+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.slo.alerts-mappings
[2024-08-20T15:12:51.791+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.uptime.alerts-mappings
[2024-08-20T15:12:51.795+00:00][INFO ][plugins.alerting] Installing component template .alerts-security.alerts-mappings
[2024-08-20T15:12:51.796+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.logs.alerts-mappings
[2024-08-20T15:12:51.797+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.metrics.alerts-mappings
[2024-08-20T15:12:51.797+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.apm.alerts-mappings
[2024-08-20T15:12:51.802+00:00][INFO ][plugins.observability] SLO resources already installed - skipping
[2024-08-20T15:12:51.804+00:00][INFO ][plugins.observabilityAIAssistant.service] Creating concrete write index - .kibana-observability-ai-assistant-conversations-000001
[2024-08-20T15:12:51.808+00:00][INFO ][plugins.alerting] Installing index template .alerts-observability.uptime.alerts-default-index-template
[2024-08-20T15:12:51.809+00:00][INFO ][plugins.alerting] Installing index template .alerts-observability.slo.alerts-default-index-template
[2024-08-20T15:12:51.810+00:00][INFO ][plugins.alerting] Installing index template .alerts-stack.alerts-default-index-template
[2024-08-20T15:12:51.828+00:00][INFO ][plugins.alerting] Installing index template .alerts-observability.logs.alerts-default-index-template
[2024-08-20T15:12:51.829+00:00][INFO ][plugins.alerting] Installing index template .alerts-observability.metrics.alerts-default-index-template
[2024-08-20T15:12:51.831+00:00][INFO ][plugins.alerting] Installing index template .alerts-observability.apm.alerts-default-index-template
[2024-08-20T15:12:51.833+00:00][INFO ][plugins.fleet] Agent policies updated by license change: []
[2024-08-20T15:12:51.834+00:00][INFO ][plugins.alerting] Installing index template .alerts-security.alerts-default-index-template
[2024-08-20T15:12:51.839+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.uptime.alerts-default-000001
[2024-08-20T15:12:51.840+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-stack.alerts-default-000001
[2024-08-20T15:12:51.841+00:00][INFO ][plugins.observability] SLO summary transforms already installed - skipping
[2024-08-20T15:12:51.854+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.slo.alerts-default-000001
[2024-08-20T15:12:51.911+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.logs.alerts-default-000001
[2024-08-20T15:12:51.913+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.metrics.alerts-default-000001
[2024-08-20T15:12:51.949+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.apm.alerts-default-000001
[2024-08-20T15:12:51.956+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-security.alerts-default-000001
[2024-08-20T15:12:52.057+00:00][ERROR][plugins.observabilityAIAssistant.service] Failed to initialize service: parse_exception
	Root causes:
		parse_exception: No processor type exists with name [inference]
[2024-08-20T15:12:52.058+00:00][ERROR][plugins.observabilityAIAssistant.service] Could not index 7 entries because of an initialisation error
[2024-08-20T15:12:52.058+00:00][ERROR][plugins.observabilityAIAssistant.service] ResponseError: parse_exception
	Root causes:
		parse_exception: No processor type exists with name [inference]
    at KibanaTransport.request (/opt/bitnami/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at KibanaTransport.request (/opt/bitnami/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at Ingest.putPipeline (/opt/bitnami/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ingest.js:139:16)
    at ObservabilityAIAssistantService.<anonymous> (/opt/bitnami/kibana/node_modules/@kbn/observability-ai-assistant-plugin/server/service/index.js:123:9)
[2024-08-20T15:12:52.530+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /opt/bitnami/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
[2024-08-20T15:12:54.949+00:00][INFO ][status] Kibana is now available (was degraded)
[2024-08-20T15:15:26.513+00:00][INFO ][root] SIGTERM received - initiating shutdown
[2024-08-20T15:15:26.514+00:00][INFO ][root] Kibana is shutting down
[2024-08-20T15:15:26.524+00:00][INFO ][plugins-system.standard] Stopping all plugins.
[2024-08-20T15:15:26.526+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
[2024-08-20T15:15:26.554+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2024-08-20T15:15:26.562+00:00][WARN ][plugins.fleet] Old fleet indices cleanup failed: There are no living connections

leandrojmp · August 20, 2024, 4:51pm

Not sure what is the issue, I don't think Kibana would crash for this error as it is not a FATAL error.

But the following lines suggests that something in your system requested a shutdown of the Kibana service and not the service itself crashing.

I do not use Kubernetes EKS or Helm, but how does it health check the service? Can it be killing it because of an error in the logs?

Glorified_SysAdmin · August 20, 2024, 5:50pm

I will have to dig through the logs. Kibana does restart once it can't find inference processor. Also, is there a reason why the node roles aren't being applied? Is there a mistake I am making?

Glorified_SysAdmin · August 20, 2024, 6:03pm

Also, ES cluster health is green

leandrojmp · August 20, 2024, 6:37pm

Unfortunately I can not help with this as I do not use Elastic and Kibana on Kubernetes, so not sure why this is not working.

Normally you just change the role of the node and restart it.

Glorified_SysAdmin · August 20, 2024, 8:44pm

How do you change the node roles normally?

leandrojmp · August 20, 2024, 9:03pm

I edit the elasticsearch.yml and restart the service in the VM.

I do not use kubernetes, nor docker container, my deployment is directly installed in the VM.

I'm not sure how this works using a heml deployment, I would assume that just changing the nodes it should change it, but you need to restart/recreate the pods.

If this is not working you need to contact the creator of the Helm file, which in your case seems to be Bitnami, as Elastic dos not publish Helm filmes anymore.

Glorified_SysAdmin · August 21, 2024, 1:28pm

Can you please refer me to documentation that references inference processor requires machine learning to be enabled? Which in turn requires a paid license?

Topic		Replies	Views
Got error "No processor type exists with name [attachment] Elasticsearch	5	1307	April 16, 2021
Ingest attachment Plugin exception : Elasticsearch	8	4423	January 19, 2017
Kibana Pod going into CrashLoopBackOff Kibana	2	807	May 14, 2020
Ingest pipeline grok processor on_failure Elasticsearch ingest-pipeline	2	819	January 4, 2021
I get the error "No processor type exists with name [attachment]" Elasticsearch	20	7794	August 5, 2019

Kibana stuck in a crash loop

Related topics