# Issues with Persistent Changes in Metricbeat Visuals and Index Mapping **URL:** \ **Category:** Kibana\ **Tags:** docker\ **Created:** [December 30, 2024, 5:45pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635 "2024-12-30T17:45:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![componentByss](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@componentByss](https://discuss.elastic.co/u/componentByss)\ **Post date:** [December 30, 2024, 5:45pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/1 "2024-12-30T17:45:09Z")
Hello everyone, I'm encountering an issue in Kibana 8.9.1 where changes made in the Visualize Library to the Metricbeat Docker ECS table, for example, do not persist after a set number of days. Specifically, the visualization defaults to a Buckets size of 5, and even though I manually change it to 100, it reverts back after a couple of days. How can I make these changes persist? Additionally, I'm seeing a similar issue with the index mapping total fields limit. After running the following command in Dev Tools: ```auto PUT /_all/_settings { "index.mapping.total_fields.limit": 15000 } ``` it reverts back to a much smaller number after a couple of weeks. Here are the relevant configuration files in case something needs to be adjusted: metricbeat.yml: ```auto metricbeat.config: modules: path: ${path.config}/modules.d/*.yml reload.enabled: false output.elasticsearch: hosts: [redacted] username: redacted password: redacted ssl.certificate_authorities: '/usr/share/elasticsearch/config/certs/ca/ca.crt' setup.kibana: host: redacted ssl.certificate_authorities: '/usr/share/elasticsearch/config/certs/ca/ca.crt' setup.dashboards.enabled: true ``` modules.d/docker.yml: ```auto - module: docker metricsets: - container - cpu - diskio - healthcheck - info - memory - network hosts: ["unix:///var/run/docker.sock"] period: 10s ``` Any insights on how to ensure these changes remain persistent would be greatly appreciated. Thank you! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2024, 6:05pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/2 "2024-12-30T18:05:59Z")
Hi @componentByss Welcome to the community. My first guess is that someone or some process is running the metricbeat setup command which is overriding your templates and visualizations dashboards.. ---
**Author:** ![componentByss](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@componentByss](https://discuss.elastic.co/u/componentByss)\ **Post date:** [December 30, 2024, 6:59pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/3 "2024-12-30T18:59:09Z")
Would having this command in the metricbeat section of the docker-compose be causing this issue? ```auto command: metricbeat -e ``` I am not seeing a metricbeat setup in any files ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2024, 7:31pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/4 "2024-12-30T19:31:12Z")
You will need to share your entire compose file.. How did you load the dashboard in the first place? > [@componentByss](#): > > Specifically, the visualization defaults to a Buckets size of 5, and even though I manually change it to 100 Can show how / what you are doing? ---
**Author:** ![componentByss](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@componentByss](https://discuss.elastic.co/u/componentByss)\ **Post date:** [December 30, 2024, 9:05pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/5 "2024-12-30T21:05:39Z")
I think the issue might be in the metricbeat.yml with setup.dashboards.enabled being set to True. I have changed it to False and will see if this is the solution for the metricbeat visuals persistence. Do you see anything in the below docker-compose that could be changed to persist the index mapping total fields limit amount? Thank you very much! Here is the docker-compose ```auto services: ${ELASTIC_COORD_NAME}: image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${IMAGE}:${VERSION} hostname: ${ELASTIC_COORD_NAME} deploy: replicas: 1 resources: limits: memory: ${MEMORY_LIMIT} volumes: - ${CERT_DIR}:${CONTAINER_CERT_DIR} - ${ELASTIC_COORD_CONF_FD}:${CONTAINER_ELASTIC_COORD_CONF_FD} - ${ELASTIC_COORD_LOGS_DIR}:${CONTAINER_LOGS_DIR} - ${SNAPSHOTS_DIR}:${CONTAINER_SNAPSHOTS_DIR} environment: - SERVER_NAME=${ELASTIC_COORD_NAME} - node.name=${ELASTIC_COORD_NAME} - cluster.name=${CLUSTER_NAME} - discovery.seed_hosts=${ELASTIC_COORD_NAME},${ELASTIC_NODE_NAME} - bootstrap.memory_lock=true - path.repo=/usr/share/elasticsearch/snapshots - "ES_JAVA_OPTS=--enable-preview" ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ ] interval: 10s timeout: 10s retries: 30 ${ELASTIC_NODE_NAME}: image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${IMAGE}:${VERSION} depends_on: - ${ELASTIC_COORD_NAME} hostname: ${ELASTIC_NODE_NAME} deploy: replicas: 1 resources: limits: memory: ${MEMORY_LIMIT} volumes: - ${CERT_DIR}:${CONTAINER_CERT_DIR} - ${DATA_DIR}:${CONTAINER_DATA_DIR} - ${LOGS_DIR}:${CONTAINER_LOGS_DIR} - ${SNAPSHOTS_DIR}:${CONTAINER_SNAPSHOTS_DIR} environment: - SERVER_NAME=${ELASTIC_NODE_NAME} - node.name=${ELASTIC_NODE_NAME} - cluster.name=${CLUSTER_NAME} - discovery.seed_hosts=${ELASTIC_COORD_NAME},${ELASTIC_NODE_NAME} - bootstrap.memory_lock=true - path.repo=/usr/share/elasticsearch/snapshots - "ES_JAVA_OPTS=--enable-preview" - thread_pool.search.queue_size=10000 ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ ] interval: 10s timeout: 10s retries: 30 ${KIBANA_NAME}: depends_on: - ${ELASTIC_NODE_NAME} image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${KIBANA_IMAGE}:${VERSION} hostname: ${KIBANA_NAME} container_name: ${KIBANA_NAME} deploy: replicas: 1 resources: limits: memory: ${MEM_LIMIT} volumes: - ${KIBANA_CERT_DIR}:${CONTAINER_KIBANA_CERT_DIR} - ${KIBANA_DATA_DIR}:${CONTAINER_KIBANA_DATA_DIR} - ${KIBANA_CONF_FD}:${CONTAINER_KIBANA_CONF_FD} environment: - SERVER_NAME=${KIBANA_NAME} - node.name=${KIBANA_NAME} - "ES_JAVA_OPTS=--enable-preview" healthcheck: test: [ ] interval: 30s timeout: 10s retries: 30 restart: 'unless-stopped' ${METRICBEAT_NAME}: depends_on: - ${KIBANA_NAME} image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${METRICBEAT_IMAGE}:${VERSION} user: root hostname: ${METRICBEAT_NAME} container_name: ${METRICBEAT_NAME} deploy: replicas: 1 resources: limits: memory: ${METRICBEAT_MEM_LIMIT} volumes: - ${CERT_DIR}:${CONTAINER_CERT_DIR} - ${DOCKER_HOST_SOCK}:${DOCKER_CONTAINER_SOCK} - ${DOCKER_HOST_LOGS}:${DOCKER_CONTAINER_LOGS} - ${METRICBEAT_CONF_FD}:${CONTAINER_METRICBEAT_CONF_FD} - ${METRICBEAT_MODULE_DIR}:${CONTAINER_METRICBEAT_MODULE_DIR} environment: - SERVER_NAME=${METRICBEAT_NAME} - node.name=${METRICBEAT_NAME} command: metricbeat -e ${FILEBEAT_NAME}: depends_on: - ${KIBANA_NAME} image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${FILEBEAT_IMAGE}:${VERSION} user: root hostname: ${FILEBEAT_NAME} container_name: ${FILEBEAT_NAME} deploy: replicas: 1 resources: limits: memory: ${MEM_LIMIT} volumes: - ${CERT_DIR}:${CONTAINER_CERT_DIR} - ${DOCKER_SOCK}:${CONTAINER_SOCK}:ro - ${LOGS_DIR}:${CONTAINER_LOGS}:ro - ${FILEBEAT_DATA}:${CONTAINER_FILEBEAT_DATA} environment: - SERVER_NAME=${FILEBEAT_NAME} - node.name=${FILEBEAT_NAME} command: filebeat -e ${LOGSTASH_NAME}: depends_on: - ${KIBANA_NAME} image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${LOGSTASH_IMAGE}:${VERSION} user: root hostname: ${LOGSTASH_NAME} container_name: ${LOGSTASH_NAME} deploy: replicas: 1 volumes: - ${CA_CERT_FD}:${CONTAINER_CA_CERT_FD} - ${CA_KEY_FD}:${CONTAINER_CA_KEY_FD} - ${LOGSTASH_CERT_FD}:${CONTAINER_LOGSTASH_CERT_FD} - ${LOGSTASH_KEY_FD}:${CONTAINER_LOGSTASH_KEY_FD} - ${DOCKER_SOCK}:${CONTAINER_SOCK}:ro - ${LOGS_DIR}:${CONTAINER_LOGS_DIR}:ro - ${SINCEDB_DIR}:${CONTAINER_SINCEDB_DIR} environment: - SERVER_NAME=${LOGSTASH_NAME} - node.name=${LOGSTASH_NAME} - "LS_JAVA_OPTS=-Xms4g -Xmx4g" command: logstash -e ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2024, 9:17pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/6 "2024-12-30T21:17:23Z")
Nice Compose! > [@componentByss](#): > > I think the issue might be in the metricbeat.yml with setup.dashboards.enabled being set to True. Yes absolutely, I saw that but was waiting [for more information](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-dashboards.html#_setup_dashboards_enabled) > When dashboard loading is enabled, Metricbeat overwrites any existing dashboards that match the names of the dashboards you are loading. This happens every time Metricbeat starts. So that is why your dashboards are being overwritten when you restart metricbeat Also... Is that your entire metricbeat.yml because what I was suspecting was that you had set see [docs](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-template.html) **`setup.template.overwrite: true`** Metricbeat will load the templates on start but should not be overwriting... unless something / another metricbeat is overwriting or explicating running setup again.. ---
**Author:** ![componentByss](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@componentByss](https://discuss.elastic.co/u/componentByss)\ **Post date:** [December 31, 2024, 1:51pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/7 "2024-12-31T13:51:42Z")
Wonderful, thank you so much for your help with the Metricbeat visuals persistence. That was the complete metricbeat.yml but good information for the next person to ensure setup.template.overwrite is set to False. For the Index Mapping limit persistence issue, I am seeing this error when checking Metricbeat docker logs: ```auto \"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [38]\"}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"} ``` I then go into Dev Tools and run ```auto PUT /_all/_settings { "index.mapping.total_fields.limit": 15000 } ``` but this reverts every few days. How can I make this persist and alleviate the need to rerun this command in Dev Tools? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 31, 2024, 3:55pm UTC](https://discuss.elastic.co/t/issues-with-persistent-changes-in-metricbeat-visuals-and-index-mapping/372635/8 "2024-12-31T15:55:25Z")
What I suspect is the following... When you run that command, it sets the settings on **existing** indexes. But when metricbeat index rolls over And creates a new index **the new index** will not have that setting You need to go into the metricbeat index template And edit it / update it and put that setting in the template. So when the next / New indexes are created, it gets that setting. By the way, This has been fixed in the latest metricbeat. It should be the same for 8.9 ![Screenshot 2024-12-31 at 8.13.18 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/5/1578b5c86426a5a3dd07a0d7c9674623ee7b0798.png)