Why does heap memory increase?

Hello.

I am curious about the increase in memory usage even though the query traffic is low and the number of documents is small, and I would like expert opinions on how to tune the GC.

[Servers (Nodes)]

  • Total: 4 Nodes (master: 1, data: 3)
  • CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (40 cores)
  • MEM: 126GB

[ElasticSearch]

  • Version: ES7.10.2 oss
  • Number of Shards: 4,
  • Number of Replicas: 1
  • The number of indexed documents is about 20,000.
  • 1 request per 1 second.
  • 20,000 doucuments are updated only at 5am. (by batch)
  • store.size: 57mb
  • Not use Kibana, Logstash, Beats
    ** Scheme
    *** about 80 fields (indexed fields: 30)
# Cluster API
{ - 
  "_nodes": { - 
    "total": 4,
    "successful": 4,
    "failed": 0
  },
  "cluster_name": "podcast1",
  "cluster_uuid": "OIlUdP3JQRuGvUpwodvnIw",
  "timestamp": 1623600113895,
  "status": "green",
  "indices": { - 
    "count": 1,
    "shards": { - 
      "total": 8,
      "primaries": 4,
      "replication": 1,
      "index": { - 
        "shards": { - 
          "min": 8,
          "max": 8,
          "avg": 8
        },
        "primaries": { - 
          "min": 4,
          "max": 4,
          "avg": 4
        },
        "replication": { - 
          "min": 1,
          "max": 1,
          "avg": 1
        }
      }
    },
    "docs": { - 
      "count": 21292,
      "deleted": 0
    },
    "store": { - 
      "size_in_bytes": 60097104,
      "reserved_in_bytes": 0
    },
    "fielddata": { - 
      "memory_size_in_bytes": 0,
      "evictions": 0
    },
    "query_cache": { - 
      "memory_size_in_bytes": 0,
      "total_count": 0,
      "hit_count": 0,
      "miss_count": 0,
      "cache_size": 0,
      "cache_count": 0,
      "evictions": 0
    },
    "completion": { - 
      "size_in_bytes": 0
    },
    "segments": { - 
      "count": 8,
      "memory_in_bytes": 128480,
      "terms_memory_in_bytes": 96384,
      "stored_fields_memory_in_bytes": 4160,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 17536,
      "points_memory_in_bytes": 0,
      "doc_values_memory_in_bytes": 10400,
      "index_writer_memory_in_bytes": 0,
      "version_map_memory_in_bytes": 0,
      "fixed_bit_set_memory_in_bytes": 0,
      "max_unsafe_auto_id_timestamp": -1,
      "file_sizes": { - 

      }
    },
    "mappings": { - 
      "field_types": [ - 
        { - 
          "name": "date",
          "count": 1,
          "index_count": 1
        },
        { - 
          "name": "integer",
          "count": 11,
          "index_count": 1
        },
        { - 
          "name": "keyword",
          "count": 21,
          "index_count": 1
        },
        { - 
          "name": "long",
          "count": 8,
          "index_count": 1
        },
        { - 
          "name": "text",
          "count": 39,
          "index_count": 1
        }
      ]
    },
    "analysis": { - 
      "char_filter_types": [ - 

      ],
      "tokenizer_types": [ - 
        { - 
          "name": "pattern",
          "count": 1,
          "index_count": 1
        }
      ],
      "filter_types": [ - 
        { - 
          "name": "pattern_replace",
          "count": 1,
          "index_count": 1
        }
      ],
      "analyzer_types": [ - 
        { - 
          "name": "custom",
          "count": 2,
          "index_count": 1
        }
      ],
      "built_in_char_filters": [ - 

      ],
      "built_in_tokenizers": [ - 
        { - 
          "name": "keyword",
          "count": 1,
          "index_count": 1
        }
      ],
      "built_in_filters": [ - 
        { - 
          "name": "lowercase",
          "count": 2,
          "index_count": 1
        }
      ],
      "built_in_analyzers": [ - 

      ]
    }
  },
  "nodes": { - 
    "count": { - 
      "total": 4,
      "coordinating_only": 0,
      "data": 4,
      "ingest": 4,
      "master": 4,
      "remote_cluster_client": 4
    },
    "versions": [ - 
      "7.10.2"
    ],
    "os": { - 
      "available_processors": 160,
      "allocated_processors": 160,
      "names": [ - 
        { - 
          "name": "Linux",
          "count": 4
        }
      ],
      "pretty_names": [ - 
        { - 
          "pretty_name": "CentOS Linux 7 (Core)",
          "count": 4
        }
      ],
      "mem": { - 
        "total_in_bytes": 269138096128,
        "free_in_bytes": 44883550208,
        "used_in_bytes": 224254545920,
        "free_percent": 17,
        "used_percent": 83
      }
    },
    "process": { - 
      "cpu": { - 
        "percent": 0
      },
      "open_file_descriptors": { - 
        "min": 360,
        "max": 368,
        "avg": 363
      }
    },
    "jvm": { - 
      "max_uptime_in_millis": 14920332,
      "versions": [ - 
        { - 
          "version": "15.0.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "15.0.1+9",
          "vm_vendor": "AdoptOpenJDK",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 4
        }
      ],
      "mem": { - 
        "heap_used_in_bytes": 9423393920,
        "heap_max_in_bytes": 68719476736
      },
      "threads": 661
    },
    "fs": { - 
      "total_in_bytes": 2397027188736,
      "free_in_bytes": 1984199778304,
      "available_in_bytes": 1984199778304
    },
    "plugins": [ - 
      { - 
        "name": "es-klp-plugin", # Korean Analyzer 
        "version": "1.12.5",
        "elasticsearch_version": "7.10.2",
        "java_version": "1.8",
        "description": "SKT klp - ES Plugin",
        "classname": "search.tech.plugin.KlpPlugin",
        "extended_plugins": [ - 

        ],
        "has_native_controller": false
      }
    ],
    "network_types": { - 
      "transport_types": { - 
        "netty4": 4
      },
      "http_types": { - 
        "netty4": 4
      }
    },
    "discovery_types": { - 
      "zen": 4
    },
    "packaging_types": [ - 
      { - 
        "flavor": "oss",
        "type": "tar",
        "count": 4
      }
    ],
    "ingest": { - 
      "number_of_pipelines": 0,
      "processor_stats": { - 

      }
    }
  }
}

[JDK and JVM options]

  • Open JDK 15 (Bundled OpenJDK)
  • Use G1GC
  • Heap:
# I didn't change anything other than the heap size.
-Xms16g -Xmx16g
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

Query requests are about 1 per second and there are almost no updates of documents.
I don't understand why the memory goes up to 60%.
(And at 5am, heap memory usage increases about 10% point by batch of updating 20000 documents)

The increase to 60% seems to be because the default value of the "-XX:G1MaxNewSizePercent" option is 60%.

I think it is a good situation that the heap memory graph is a saw feature,
But I hope that the slope of the graph decreases, resulting in a gradual memory rise and a short gc.

What should I do?

Please upgrade, the latest is 7.13 and there's always improvements being made around this topic.

You really need only 1 shard and 1 replica for this dataset. If you want the data on all nodes, then increase the replica count, not the primary shard count.

Nothing, there's nothing here that suggests you have any problems and Elasticsearch and the JVM appear to be running as expected.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.