Issues with Persistent Changes in Metricbeat Visuals and Index Mapping

componentByss · December 30, 2024, 5:45pm

Hello everyone,

I'm encountering an issue in Kibana 8.9.1 where changes made in the Visualize Library to the Metricbeat Docker ECS table, for example, do not persist after a set number of days. Specifically, the visualization defaults to a Buckets size of 5, and even though I manually change it to 100, it reverts back after a couple of days. How can I make these changes persist?

Additionally, I'm seeing a similar issue with the index mapping total fields limit. After running the following command in Dev Tools:

PUT /_all/_settings
{
  "index.mapping.total_fields.limit": 15000
}

it reverts back to a much smaller number after a couple of weeks.

Here are the relevant configuration files in case something needs to be adjusted:

metricbeat.yml:

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

output.elasticsearch:
  hosts:    [redacted]
  username: redacted
  password: redacted
  ssl.certificate_authorities: '/usr/share/elasticsearch/config/certs/ca/ca.crt'

setup.kibana:
  host: redacted
  ssl.certificate_authorities: '/usr/share/elasticsearch/config/certs/ca/ca.crt'

setup.dashboards.enabled: true

modules.d/docker.yml:

- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s

Any insights on how to ensure these changes remain persistent would be greatly appreciated. Thank you!

stephenb · December 30, 2024, 6:05pm

Hi @componentByss

Welcome to the community.

My first guess is that someone or some process is running the metricbeat setup command which is overriding your templates and visualizations dashboards..

componentByss · December 30, 2024, 6:59pm

Would having this command in the metricbeat section of the docker-compose be causing this issue?

    command: metricbeat -e

I am not seeing a metricbeat setup in any files

stephenb · December 30, 2024, 7:31pm

You will need to share your entire compose file..

How did you load the dashboard in the first place?

Can show how / what you are doing?

componentByss · December 30, 2024, 9:05pm

I think the issue might be in the metricbeat.yml with setup.dashboards.enabled being set to True. I have changed it to False and will see if this is the solution for the metricbeat visuals persistence. Do you see anything in the below docker-compose that could be changed to persist the index mapping total fields limit amount? Thank you very much!

Here is the docker-compose

services:
  ${ELASTIC_COORD_NAME}:
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${IMAGE}:${VERSION}
    hostname: ${ELASTIC_COORD_NAME}
    deploy:
      replicas: 1
      resources:
        limits:
          memory: ${MEMORY_LIMIT}
    volumes:
      - ${CERT_DIR}:${CONTAINER_CERT_DIR}
      - ${ELASTIC_COORD_CONF_FD}:${CONTAINER_ELASTIC_COORD_CONF_FD}
      - ${ELASTIC_COORD_LOGS_DIR}:${CONTAINER_LOGS_DIR}
      - ${SNAPSHOTS_DIR}:${CONTAINER_SNAPSHOTS_DIR}
    environment:
      - SERVER_NAME=${ELASTIC_COORD_NAME}
      - node.name=${ELASTIC_COORD_NAME}
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=${ELASTIC_COORD_NAME},${ELASTIC_NODE_NAME}
      - bootstrap.memory_lock=true
      - path.repo=/usr/share/elasticsearch/snapshots
      - "ES_JAVA_OPTS=--enable-preview"

    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
        ]
      interval: 10s
      timeout: 10s
      retries: 30

  ${ELASTIC_NODE_NAME}:
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${IMAGE}:${VERSION}
    depends_on:
      - ${ELASTIC_COORD_NAME}
    hostname: ${ELASTIC_NODE_NAME}
    deploy:
      replicas: 1
      resources:
        limits:
          memory: ${MEMORY_LIMIT}
    volumes:
      - ${CERT_DIR}:${CONTAINER_CERT_DIR}
      - ${DATA_DIR}:${CONTAINER_DATA_DIR}
      - ${LOGS_DIR}:${CONTAINER_LOGS_DIR}
      - ${SNAPSHOTS_DIR}:${CONTAINER_SNAPSHOTS_DIR}
    environment:
      - SERVER_NAME=${ELASTIC_NODE_NAME}
      - node.name=${ELASTIC_NODE_NAME}
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=${ELASTIC_COORD_NAME},${ELASTIC_NODE_NAME}
      - bootstrap.memory_lock=true
      - path.repo=/usr/share/elasticsearch/snapshots
      - "ES_JAVA_OPTS=--enable-preview"
      - thread_pool.search.queue_size=10000

    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [ 
        ]
      interval: 10s
      timeout: 10s
      retries: 30

  ${KIBANA_NAME}:
    depends_on:
      - ${ELASTIC_NODE_NAME}
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${KIBANA_IMAGE}:${VERSION}
    hostname: ${KIBANA_NAME}
    container_name: ${KIBANA_NAME}
    deploy:
      replicas: 1
      resources:
        limits:
          memory: ${MEM_LIMIT}
    volumes:
      - ${KIBANA_CERT_DIR}:${CONTAINER_KIBANA_CERT_DIR}
      - ${KIBANA_DATA_DIR}:${CONTAINER_KIBANA_DATA_DIR}
      - ${KIBANA_CONF_FD}:${CONTAINER_KIBANA_CONF_FD}
    environment:
      - SERVER_NAME=${KIBANA_NAME}
      - node.name=${KIBANA_NAME}
      - "ES_JAVA_OPTS=--enable-preview"

    healthcheck:
      test:
        [
        ]
      interval: 30s
      timeout: 10s
      retries: 30
    restart: 'unless-stopped'

  ${METRICBEAT_NAME}:
    depends_on:
      - ${KIBANA_NAME}
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${METRICBEAT_IMAGE}:${VERSION}
    user: root
    hostname: ${METRICBEAT_NAME}
    container_name: ${METRICBEAT_NAME}
    deploy:
      replicas: 1
      resources:
        limits:
          memory: ${METRICBEAT_MEM_LIMIT}
    volumes:
      - ${CERT_DIR}:${CONTAINER_CERT_DIR}
      - ${DOCKER_HOST_SOCK}:${DOCKER_CONTAINER_SOCK}
      - ${DOCKER_HOST_LOGS}:${DOCKER_CONTAINER_LOGS}
      - ${METRICBEAT_CONF_FD}:${CONTAINER_METRICBEAT_CONF_FD}
      - ${METRICBEAT_MODULE_DIR}:${CONTAINER_METRICBEAT_MODULE_DIR}
    environment:
      - SERVER_NAME=${METRICBEAT_NAME}
      - node.name=${METRICBEAT_NAME}
    command: metricbeat -e

  ${FILEBEAT_NAME}:
    depends_on:
      - ${KIBANA_NAME}
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${FILEBEAT_IMAGE}:${VERSION}
    user: root    
    hostname: ${FILEBEAT_NAME}
    container_name: ${FILEBEAT_NAME}
    deploy:
      replicas: 1
      resources:
        limits:
          memory: ${MEM_LIMIT}
    volumes:
      - ${CERT_DIR}:${CONTAINER_CERT_DIR}
      - ${DOCKER_SOCK}:${CONTAINER_SOCK}:ro
      - ${LOGS_DIR}:${CONTAINER_LOGS}:ro
      - ${FILEBEAT_DATA}:${CONTAINER_FILEBEAT_DATA}
    environment:
      - SERVER_NAME=${FILEBEAT_NAME}
      - node.name=${FILEBEAT_NAME}
    command: filebeat -e
   
  ${LOGSTASH_NAME}:
    depends_on:
      - ${KIBANA_NAME}
    image: ${REGISTRY_HOST}:${REGISTRY_PORT}/${LOGSTASH_IMAGE}:${VERSION}
    user: root
    hostname: ${LOGSTASH_NAME}
    container_name: ${LOGSTASH_NAME}
    deploy:
      replicas: 1
    volumes:
      - ${CA_CERT_FD}:${CONTAINER_CA_CERT_FD}
      - ${CA_KEY_FD}:${CONTAINER_CA_KEY_FD}
      - ${LOGSTASH_CERT_FD}:${CONTAINER_LOGSTASH_CERT_FD}
      - ${LOGSTASH_KEY_FD}:${CONTAINER_LOGSTASH_KEY_FD}
      - ${DOCKER_SOCK}:${CONTAINER_SOCK}:ro
      - ${LOGS_DIR}:${CONTAINER_LOGS_DIR}:ro
      - ${SINCEDB_DIR}:${CONTAINER_SINCEDB_DIR}
    environment:
      - SERVER_NAME=${LOGSTASH_NAME}
      - node.name=${LOGSTASH_NAME}
      - "LS_JAVA_OPTS=-Xms4g -Xmx4g"
    command: logstash -e

stephenb · December 30, 2024, 9:17pm

Nice Compose!

Yes absolutely, I saw that but was waiting for more information

When dashboard loading is enabled, Metricbeat overwrites any existing dashboards that match the names of the dashboards you are loading. This happens every time Metricbeat starts.

So that is why your dashboards are being overwritten when you restart metricbeat

Also...

Is that your entire metricbeat.yml because what I was suspecting was that you had set see docs

setup.template.overwrite: true

Metricbeat will load the templates on start but should not be overwriting... unless something / another metricbeat is overwriting or explicating running setup again..

componentByss · December 31, 2024, 1:51pm

Wonderful, thank you so much for your help with the Metricbeat visuals persistence. That was the complete metricbeat.yml but good information for the next person to ensure setup.template.overwrite is set to False.

For the Index Mapping limit persistence issue, I am seeing this error when checking Metricbeat docker logs:

\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [38]\"}}, dropping event!","service.name":"metricbeat","ecs.version":"1.6.0"}

I then go into Dev Tools and run

 PUT /_all/_settings
{
  "index.mapping.total_fields.limit": 15000
}

but this reverts every few days. How can I make this persist and alleviate the need to rerun this command in Dev Tools?

stephenb · December 31, 2024, 3:55pm

What I suspect is the following...

When you run that command, it sets the settings on existing indexes.
But when metricbeat index rolls over And creates a new index the new index will not have that setting

You need to go into the metricbeat index template And edit it / update it and put that setting in the template. So when the next / New indexes are created, it gets that setting.

By the way, This has been fixed in the latest metricbeat.

It should be the same for 8.9

Topic		Replies	Views
Kibana index settings and dashboards are always lost when updating elk container Kibana docker	3	1300	March 19, 2019
Missing fields when viewing dashboard in Kibana Beats	2	930	November 1, 2017
Metricbeat not sending data after recreating index Beats metricbeat	1	505	January 1, 2021
Problem with visualization in [Metricbeat System] Host overview Kibana	3	408	July 10, 2018
Kibana loses visualization edits? Kibana	3	919	November 6, 2020

Issues with Persistent Changes in Metricbeat Visuals and Index Mapping

Related topics