# Kibana error on migration saved-object

**URL:** <https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098>\
**Category:** Kibana\
**Created:** [September 13, 2021, 3:41pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098 "2021-09-13T15:41:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 3:41pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/1 "2021-09-13T15:41:52Z")

</div>

How can I add memory on kibana, for solving problem on saved\_object? I create more than one topic but I've not received any response.

---

<div class="post-metadata">

**Author:** ![corey.robertson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/corey.robertson/32/54611_2.png) [@corey.robertson](https://discuss.elastic.co/u/corey.robertson)\
**Post date:** [September 13, 2021, 3:51pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/2 "2021-09-13T15:51:50Z")

</div>

Hi @LombardoAndrea195

Can you expand a little more about what the problem is. Is your Kibana instance running out of memory during upgrade? How are you running your stack? (Docker? On the Cloud? On a local machine?)

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 3:55pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/3 "2021-09-13T15:55:51Z")

</div>

Yes of course. I push data into elasticsearch using the routines on elasticsearchpy.More in detail I use index routine for create the first document with the first index and then I push data using create. The data is pushed through kafka. So I recive a document at time. The amount of total data is not more than 250MB.Althought I insert all of this data and create some saved\_object using altair and eland as suggested in the [link](https://www.elastic.co/blog/how-to-jupyter-notebook-visualizations-kibana-dashboards-vega-data-science) for creating panels so saved object for dashboards where I can see the evolution of a value in time.

```auto

version: "2"

services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "a:1:1,b:1:1,c:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
         
  jobmanager:
    image: pyflink/playgrounds:1.10.0
    volumes:
      - ./examples:/opt/examples
    hostname: "jobmanager"
    expose:
      - "6123"
    ports:
      - "8088:8088"
    command: jobmanager
    environment:
     - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
  taskmanager:
    image: pyflink/playgrounds:1.10.0
    volumes:
      - ./examples:/opt/examples
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - jobmanager:jobmanager
    environment:
    - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
  
  elasticsearch0:
    restart: always
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
    container_name: elasticsearch
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data    
    ports:
      - 9200:9200
    networks:
      - es-net
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS:"-Xms1g-Xmx1g"
      - refresh_rate:300s
      - MALLOCA_ARENA_MAX=4
  
  kibana:
    image: docker.elastic.co/kibana/kibana:7.14.1
    mem_reservation: 9096m
    mem_limit: 9096m
    container_name: kibana
    networks:
      - es-net
    environment:
      ELASTICSEARCH_HOSTS: "http://elasticsearch:9200"     
      server.maxPayload: 1653247624
      NODE_OPTIONS=--max-old-space-size: 4048
      migrations.batchSize: 5000
      savedObjects.maxImportExportSize: 2000
      migrations.retryAttempts: 20
      savedObjects.maxImportPayloadBytes: 26214400
      mem_limit: 7096m
      mem_reservation: 8096m  

      
    ports:
      - 5601:5601
    links:

      - elasticsearch0

volumes:
  data01:
    driver: local
networks:
  es-net:
    driver: bridge

```

above there is the docker file

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 3:59pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/4 "2021-09-13T15:59:31Z")

</div>

Error generated in Kibana log is the following:

```auto
{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 43ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 387ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 450ms."}

{"type":"log","@timestamp":"2021-09-11T11:08:43+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana_task_manager] Migration completed after 1800ms"}

{"type":"log","@timestamp":"2021-09-11T11:08:47+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] Starting to process 19 documents."}

{"type":"log","@timestamp":"2021-09-11T11:08:47+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 5340ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:14+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 26286ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:17+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 3252ms."}

{"type":"log","@timestamp":"2021-09-11T11:09:17+00:00","tags":["error","savedobjects-service"],"pid":1214,"message":"[.kibana] migration failed, dumping execution log:"}

```

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 4:03pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/5 "2021-09-13T16:03:08Z")

</div>

I don't what happens I change the version from 12.1 to 14.1 and everything goes well when I start inserting data pushing it with kafka. AS I said before I have some saved\_object saved but I don't what happens. The only thing that I know is that in browser compares OUT of memory and kibana crush with exit 1. From that situation system is not able to respond and kibana arrive at the step to appear kibana server is not ready yet and then after a while everything crush. I think that I miss some configuration issue because data is too short in terms of quantity

---

<div class="post-metadata">

**Author:** ![corey.robertson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/corey.robertson/32/54611_2.png) [@corey.robertson](https://discuss.elastic.co/u/corey.robertson)\
**Post date:** [September 13, 2021, 4:20pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/6 "2021-09-13T16:20:59Z")

</div>

So after upgrading, everything is good, it's only when you're attempting to view a dashboard that causes a crash? Am I understanding that correctly?

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 4:45pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/7 "2021-09-13T16:45:26Z")

</div>

Mmm not exactly. The problem is when I insert data into elasticsearch and I've into kibana also some saved\_object. I think that the saved object occupy some weights in the server. If I insert all data into elastic everything goes well but I shouldn't create any saved object for the dashboards.

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 13, 2021, 5:02pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/8 "2021-09-13T17:02:16Z")

</div>

i don't if it could be an help for understnading the error:

```auto
{"type":"log","@timestamp":"2021-09-13T16:57:19+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana_task_manager] Migration completed after 1409ms"}

{"type":"log","@timestamp":"2021-09-13T16:57:22+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] Starting to process 19 documents."}

{"type":"log","@timestamp":"2021-09-13T16:57:22+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 4365ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:13+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 51088ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:16+00:00","tags":["info","savedobjects-service"],"pid":1213,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 3132ms."}

{"type":"log","@timestamp":"2021-09-13T16:58:16+00:00","tags":["error","savedobjects-service"],"pid":1213,"message":"[.kibana] migration failed, dumping execution log:"}

```

after this the system start crushing

---

<div class="post-metadata">

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\
**Post date:** [September 14, 2021, 2:18pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/9 "2021-09-14T14:18:26Z")

</div>

Please share all the Kibana logs during the upgrade.

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 14, 2021, 2:19pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/10 "2021-09-14T14:19:37Z")

</div>

there are too much information and they doesn't enter into the section how can I do it?

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 14, 2021, 2:22pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/11 "2021-09-14T14:22:33Z")

</div>

Can you help me on that?

---

<div class="post-metadata">

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\
**Post date:** [September 15, 2021, 7:43am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/12 "2021-09-15T07:43:07Z")

</div>

You can drag and drop attachment files into your reply

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 15, 2021, 7:57am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/13 "2021-09-15T07:57:32Z")

</div>

With drag and drop file system reply that doesn't support files different from jpeg,png of gif file I'll try to do in a different manner

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 15, 2021, 7:59am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/14 "2021-09-15T07:59:49Z")

</div>

I've inserted the file into a repository of github in order to give you the possibility to see all the log [link](https://github.com/LombardoAndrea195/ShareYourSport/blob/master/log%20kibana.txt)

---

<div class="post-metadata">

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\
**Post date:** [September 15, 2021, 9:08am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/15 "2021-09-15T09:08:25Z")

</div>

Are you sure these are all the logs? I would expect to see a log entry with `tags: ["fatal"]`

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 15, 2021, 9:28am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/16 "2021-09-15T09:28:54Z")

</div>

yes it's coming from kibana. Do you need also what is coming from log elastic?

What is present with Fatal is this one in kibana

```auto
{"type":"log","@timestamp":"2021-09-14T22:30:54+00:00","tags":["info","savedobjects-service"],"pid":1214,"message":"[.kibana] TRANSFORMED_DOCUMENTS_BULK_INDEX -> FATAL. took: 9596ms."}

```

---

<div class="post-metadata">

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\
**Post date:** [September 15, 2021, 11:45am UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/17 "2021-09-15T11:45:52Z")

</div>

When migrations fail the last log line should be something like

```auto
{... "tags":["fatal"], "message": "Unable to complete saved object migrations for the [.kibana] index. Error: [a stack trace]"}

```

Without this it's hard to know why it's failing other than that it fails during the `TRANSFORMED_DOCUMENTS_BULK_INDEX` step.

Could you try to run Kibana with debug logging enabled, I think adding `LOGGING_ROOT_LEVEL='all'` to the docker `environment` property should do that.

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 15, 2021, 12:14pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/18 "2021-09-15T12:14:25Z")

</div>

> [@rudolf](#):
>
> LOGGING\_ROOT\_LEVEL='all'

I don't know if i do everything well: because I'm a beginner also in docker. I use the shape as LOGGING\_ROOT\_LEVEL: 'all'. I share with you the outcome of the log file. I think that the problem is not more related on migration on the version but is related on migration of the saved object. [link](https://github.com/LombardoAndrea195/ShareYourSport/blob/master/log_kibana2.txt).

---

<div class="post-metadata">

**Author:** ![rudolf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rudolf/32/51200_2.png) [@rudolf](https://discuss.elastic.co/u/rudolf)\
**Post date:** [September 15, 2021, 2:53pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/19 "2021-09-15T14:53:32Z")

</div>

Sorry, I should have double checked that... Using `LOGGING_VERBOSE=true` should give you verbose logging and you'll see log records with `..."tags": ["debug", ...`

---

<div class="post-metadata">

**Author:** ![LombardoAndrea195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lombardoandrea195/32/88080_2.png) [@LombardoAndrea195](https://discuss.elastic.co/u/LombardoAndrea195)\
**Post date:** [September 15, 2021, 5:46pm UTC](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098/20 "2021-09-15T17:46:35Z")

</div>

sorry for replying you in late [link](https://raw.githubusercontent.com/LombardoAndrea195/ShareYourSport/master/kibana%20verbose)

[Next page](https://discuss.elastic.co/t/kibana-error-on-migration-saved-object/284098.md?page=2)