Java Heap Space

Elastic Stack Logstash
1

Hi,
I am getting OutOfMemoryError following are the JVM sizes
> intx CompilerThreadStackSize = 0 {pd product}
> uintx ErgoHeapSizeLimit = 0 {product}
> uintx HeapSizePerGCThread = 87241520 {product}
> uintx InitialHeapSize := 260046848 {product}
> uintx LargePageHeapSizeThreshold = 134217728 {product}
> uintx MaxHeapSize := 4158652416 {product}
> intx ThreadStackSize = 1024 {pd product}
> intx VMThreadStackSize = 1024 {pd product}

I am running logstash with multiple pipeline. I have three logstash conf file. Each will fetch data from mysql database and put in Elasticsearch.

Data is huge its like a 40 millions record for each conf file fetching. let me share my pipelines.yml file.

  • pipeline.id: morph_styles

path.config: "/home/ubuntu/dev/V4QaShop/logstash/morph_styles.conf"
pipeline.workers: 1

  • pipeline.id: demo_morph_styles
    path.config: "/home/ubuntu/dev/V4QaShop/logstash/demo_morph_styles.conf"
    pipeline.workers: 3
  • pipeline.id: product_detail
    path.config: "/home/ubuntu/dev/V4QaShop/logstash/products_detail.conf"
    queue.type: persisted

every config file schedule to 10 sec but its delay that I can not effort data will be sync in every second. What should I do I need help.

Thanks

2

Enable a heap dump when you run out of memory. Analyze the heap dump with a memory analyzer (personally I like Eclipse MAT). Tell us what the leak candidates are, what OS you are on, and what JVM you are using (java -version).

3

Following are the detail of my server

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

Java version is

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~16.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

I am looking for heap dump unable to find meaning full information

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.