Increase the replica size for the index

Elastic Stack Elasticsearch
1

Hi Team,

we are continously searching the records from the particular index approxmately 25,000 times per hour,so the heap usage is increasing regular period.

we already set the replica value as 1. Now we planned to increase the replica to 2 , to reduce the search load for that particular index.

Is this really helpful to distribute the laod across the data nodes.

2

Is heap usage a problem or is it increasing and decreasing normally? Do you see any evidence of frequent or long GC in the Elasticsearch logs? Do you have monitoring installed so we can see how heap usage varies over time?

It can be as there are more shards that can serve requests and spread the load across more nodes.

How many nodes do you have in the cluster? How many primary and replica shards does the index you are referring to have? What is the size and specification of the cluster?

Which version of Elasticsearch are you using?

3

we have faced the GC issues in the cluster.
[o.e.m.j.JvmGcMonitorService] [node1] [gc][144177] overhead, spent [1.7s] collecting in the last [2s]

Master nodes : 3
Data nodes: 5
No.of.shard for the index: 5 shards and 1 replica
heap allocated for the Each node : 10GB
size of the cluster: 200GB
Elastic version: 7.3.2

4

That is a very old version that has been EOL for a long time. I would recommend you upgrade to at least 7.17.

This means that each node already probably hold 2 shards for that index if it is spread out correctly. If that is the case I do not think adding a replica will help as it likely will increase increase heap usage.

I seems like you may need to increase the heap size or try to improve heap usage. Are there other indices in the cluster? What is the size of the index you are talking about here?

5

for that index it has the primary data of 6.1mb. It has doc count of 9920.

-XX:CMSInitiatingOccupancyFraction=75 for that once it reaches 75% of heap usage, automatically GC occurs.Is this any other possibility to reduce the GC.

6

What is the full output of the cluster stats API?

7 
{
  "_nodes": {
    "total": 8,
    "successful": 8,
    "failed": 0
  },
  "cluster_name": "cluster",
  "cluster_uuid": "ZN8eyX57QDO56ovr3y8mvA",
  "timestamp": 1688454625658,
  "status": "green",
  "indices": {
    "count": 41,
    "shards": {
      "total": 378,
      "primaries": 189,
      "replication": 1.0,
      "index": {
        "shards": {
          "min": 2,
          "max": 10,
          "avg": 9.21951219512195
        },
        "primaries": {
          "min": 1,
          "max": 5,
          "avg": 4.609756097560975
        },
        "replication": {
          "min": 1.0,
          "max": 1.0,
          "avg": 1.0
        }
      }
    },
    "docs": {
      "count": 68611649,
      "deleted": 1915444
    },
    "store": {
      "size_in_bytes": 449040832933
    },
    "fielddata": {
      "memory_size_in_bytes": 0,
      "evictions": 0
    },
    "query_cache": {
      "memory_size_in_bytes": 727190,
      "total_count": 8033514,
      "hit_count": 3456947,
      "miss_count": 4576567,
      "cache_size": 151,
      "cache_count": 100263,
      "evictions": 100112
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 2324,
      "memory_in_bytes": 209140907,
      "terms_memory_in_bytes": 39222468,
      "stored_fields_memory_in_bytes": 163657096,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 46464,
      "points_memory_in_bytes": 4500333,
      "doc_values_memory_in_bytes": 1714546,
      "index_writer_memory_in_bytes": 6503520,
      "version_map_memory_in_bytes": 950,
      "fixed_bit_set_memory_in_bytes": 8358160,
      "max_unsafe_auto_id_timestamp": 1688404041485,
      "file_sizes": {
        
      }
    }
  },
  "nodes": {
    "count": {
      "total": 8,
      "coordinating_only": 0,
      "data": 5,
      "ingest": 0,
      "master": 3,
      "voting_only": 0
    },
    "versions": [
      "7.3.2"
    ],
    "os": {
      "available_processors": 106,
      "allocated_processors": 106,
      "names": [
        {
          "name": "Linux",
          "count": 8
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Ubuntu 20.04.6 LTS",
          "count": 8
        }
      ],
      "mem": {
        "total_in_bytes": 885082402816,
        "free_in_bytes": 155269054464,
        "used_in_bytes": 729813348352,
        "free_percent": 18,
        "used_percent": 82
      }
    },
    "process": {
      "cpu": {
        "percent": 5
      },
      "open_file_descriptors": {
        "min": 560,
        "max": 7534,
        "avg": 2946
      }
    },
    "jvm": {
      "max_uptime_in_millis": 6555932582,
      "versions": [
        {
          "version": "1.8.0_131",
          "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version": "25.131-b11",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": false,
          "count": 8
        }
      ],
      "mem": {
        "heap_used_in_bytes": 25887638752,
        "heap_max_in_bytes": 60699967488
      },
      "threads": 1046
    },
    "fs": {
      "total_in_bytes": 3081931849728,
      "free_in_bytes": 1080494608384,
      "available_in_bytes": 1080393945088
    },
    "plugins": [
      {
        "name": "repository-gcs",
        "version": "7.3.2",
        "elasticsearch_version": "7.3.2",
        "java_version": "1.8",
        "description": "The GCS repository plugin adds Google Cloud Storage support for repositories.",
        "classname": "org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin",
        "extended_plugins": [
          
        ],
        "has_native_controller": false
      }
    ],
    "network_types": {
      "transport_types": {
        "security4": 8
      },
      "http_types": {
        "security4": 8
      }
    },
    "discovery_types": {
      "zen": 8
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "tar",
        "count": 8
      }
    ]
  }
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.