Kibana error on migration saved-object

How can I add memory on kibana, for solving problem on saved_object? I create more than one topic but I've not received any response.

Hi @LombardoAndrea195

Can you expand a little more about what the problem is. Is your Kibana instance running out of memory during upgrade? How are you running your stack? (Docker? On the Cloud? On a local machine?)

Yes of course. I push data into elasticsearch using the routines on elasticsearchpy.More in detail I use index routine for create the first document with the first index and then I push data using create. The data is pushed through kafka. So I recive a document at time. The amount of total data is not more than 250MB.Althought I insert all of this data and create some saved_object using altair and eland as suggested in the link for creating panels so saved object for dashboards where I can see the evolution of a value in time.

version: "2"

    image: wurstmeister/zookeeper:3.4.6
      - "2181:2181"
    image: wurstmeister/kafka
      - "9092:9092"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "a:1:1,b:1:1,c:1:1"
      - /var/run/docker.sock:/var/run/docker.sock
    image: pyflink/playgrounds:1.10.0
      - ./examples:/opt/examples
    hostname: "jobmanager"
      - "6123"
      - "8088:8088"
    command: jobmanager
     - |
        jobmanager.rpc.address: jobmanager
    image: pyflink/playgrounds:1.10.0
      - ./examples:/opt/examples
      - "6121"
      - "6122"
      - jobmanager
    command: taskmanager
      - jobmanager:jobmanager
    - |
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
    restart: always
    container_name: elasticsearch
        soft: -1
        hard: -1
      - data01:/usr/share/elasticsearch/data    
      - 9200:9200
      - es-net
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS:"-Xms1g-Xmx1g"
      - refresh_rate:300s
    mem_reservation: 9096m
    mem_limit: 9096m
    container_name: kibana
      - es-net
      ELASTICSEARCH_HOSTS: "http://elasticsearch:9200"     
      server.maxPayload: 1653247624
      NODE_OPTIONS=--max-old-space-size: 4048
      migrations.batchSize: 5000
      savedObjects.maxImportExportSize: 2000
      migrations.retryAttempts: 20
      savedObjects.maxImportPayloadBytes: 26214400
      mem_limit: 7096m
      mem_reservation: 8096m  

      - 5601:5601

      - elasticsearch0

    driver: local
    driver: bridge

above there is the docker file

Error generated in Kibana log is the following:

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 43ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 387ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 450ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] Migration completed after 1800ms"}

{"type":"log","@timestamp":"2021-09-11T11:08:47+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] Starting to process 19 documents."}

{"type":"log","@timestamp":"2021-09-11T11:08:47+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 5340ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:14+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 26286ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:17+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 3252ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:17+00:00","tags":["error","savedobjects-service"],"pid":1214,"message":"[.kibana] migration failed, dumping execution log:"}

I don't what happens I change the version from 12.1 to 14.1 and everything goes well when I start inserting data pushing it with kafka. AS I said before I have some saved_object saved but I don't what happens. The only thing that I know is that in browser compares OUT of memory and kibana crush with exit 1. From that situation system is not able to respond and kibana arrive at the step to appear kibana server is not ready yet and then after a while everything crush. I think that I miss some configuration issue because data is too short in terms of quantity

So after upgrading, everything is good, it's only when you're attempting to view a dashboard that causes a crash? Am I understanding that correctly?

Mmm not exactly. The problem is when I insert data into elasticsearch and I've into kibana also some saved_object. I think that the saved object occupy some weights in the server. If I insert all data into elastic everything goes well but I shouldn't create any saved object for the dashboards.

i don't if it could be an help for understnading the error:

{"type":"log","@timestamp":"2021-09-13T16:57:19+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana_task_manager] Migration completed after 1409ms"}

{"type":"log","@timestamp":"2021-09-13T16:57:22+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] Starting to process 19 documents."}

{"type":"log","@timestamp":"2021-09-13T16:57:22+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 4365ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:13+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 51088ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:16+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 3132ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:16+00:00","tags":["error","savedobjects-service"],"pid":1213,"message":"[.kibana] migration failed, dumping execution log:"}

after this the system start crushing

Please share all the Kibana logs during the upgrade.

there are too much information and they doesn't enter into the section how can I do it?

Can you help me on that?

You can drag and drop attachment files into your reply

With drag and drop file system reply that doesn't support files different from jpeg,png of gif file I'll try to do in a different manner

I've inserted the file into a repository of github in order to give you the possibility to see all the log link

Are you sure these are all the logs? I would expect to see a log entry with tags: ["fatal"]

yes it's coming from kibana. Do you need also what is coming from log elastic?

What is present with Fatal is this one in kibana

{"type":"log","@timestamp":"2021-09-14T22:30:54+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 9596ms."}

When migrations fail the last log line should be something like

{... "tags":["fatal"], "message": "Unable to complete saved object migrations for the [.kibana] index. Error: [a stack trace]"}

Without this it's hard to know why it's failing other than that it fails during the TRANSFORMED_DOCUMENTS_BULK_INDEX step.

Could you try to run Kibana with debug logging enabled, I think adding LOGGING_ROOT_LEVEL='all' to the docker environment property should do that.

I don't know if i do everything well: because I'm a beginner also in docker. I use the shape as LOGGING_ROOT_LEVEL: 'all'. I share with you the outcome of the log file. I think that the problem is not more related on migration on the version but is related on migration of the saved object. link.

Sorry, I should have double checked that... Using LOGGING_VERBOSE=true should give you verbose logging and you'll see log records with ..."tags": ["debug", ...

sorry for replying you in late link