Not able to get rid of circuit breaking exception

Elastic Stack Elasticsearch
1

Hi all,
I am trying to index a 6 GB file to my cluster of three nodes. But i get circuit breaking exception every time.
I have tried this increasing the limit by,
"persistent" : { "indices.breaker.fielddata.limit" : "95%", "indices.breaker.total.limit" : "95%" }
but no effect.

I use a python script to index the documents and this is the message that i get from python library.

Traceback (most recent call last): File "json_array.py", line 34, in <module> deque(pb, maxlen=0) File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/actions.py", line 365, in parallel_bulk actions, chunk_size, max_chunk_bytes, client.transport.serializer File "/usr/lib64/python2.7/multiprocessing/pool.py", line 655, in next raise value elasticsearch.exceptions.TransportError: TransportError(429, u'circuit_breaking_exception', u'[parent] Data too large, data for [<http_request>] would be [1749216548/1.6gb], which is larger than the limit of [1717986918/1.5gb], real usage: [1748416424/1.6gb], new bytes reserved: [800124/781.3kb]')

You can find elastic logs in this link

https://we.tl/t-uPQqBulwUN

can you guys suggest anything, i am stuck here for days.

Thanks

2

How large are your bulk requests?

3

i have set chunk size as 50
i have played around with it. but the result is the same

4

How large are your documents?

5

Look at my elastic logs shared via link .
do you think the mapping is not properly done.

6

approximately 20kb per document but it varies

7

What is the full output of the cluster stats API?

8 
{
    "_nodes": {
        "total": 3,
        "successful": 3,
        "failed": 0
    },
    "cluster_name": "Elastic-Demo",
    "cluster_uuid": "XXThKV0TQN2yTWPiM3wEiw",
    "timestamp": 1566450358927,
    "status": "green",
    "indices": {
        "count": 5,
        "shards": {
            "total": 15,
            "primaries": 9,
            "replication": 0.6666666666666666,
            "index": {
                "shards": {
                    "min": 2,
                    "max": 6,
                    "avg": 3.0
                },
                "primaries": {
                    "min": 1,
                    "max": 3,
                    "avg": 1.8
                },
                "replication": {
                    "min": 0.0,
                    "max": 1.0,
                    "avg": 0.8
                }
            }
        },
        "docs": {
            "count": 465656,
            "deleted": 22
        },
        "store": {
            "size_in_bytes": 2891931250
        },
        "fielddata": {
            "memory_size_in_bytes": 0,
            "evictions": 0
        },
        "query_cache": {
            "memory_size_in_bytes": 0,
            "total_count": 9,
            "hit_count": 0,
            "miss_count": 9,
            "cache_size": 0,
            "cache_count": 0,
            "evictions": 0
        },
        "completion": {
            "size_in_bytes": 0
        },
        "segments": {
            "count": 111,
            "memory_in_bytes": 10668772,
            "terms_memory_in_bytes": 9323197,
            "stored_fields_memory_in_bytes": 594488,
            "term_vectors_memory_in_bytes": 0,
            "norms_memory_in_bytes": 677888,
            "points_memory_in_bytes": 18163,
            "doc_values_memory_in_bytes": 55036,
            "index_writer_memory_in_bytes": 0,
            "version_map_memory_in_bytes": 0,
            "fixed_bit_set_memory_in_bytes": 0,
            "max_unsafe_auto_id_timestamp": -1,
            "file_sizes": {}
        }
    },
    "nodes": {
        "count": {
            "total": 3,
            "data": 3,
            "coordinating_only": 0,
            "master": 3,
            "ingest": 3
        },
        "versions": [
            "7.1.1"
        ],
        "os": {
            "available_processors": 6,
            "allocated_processors": 6,
            "names": [
                {
                    "name": "Linux",
                    "count": 3
                }
            ],
            "pretty_names": [
                {
                    "pretty_name": "Red Hat Enterprise Linux Server 7.6 (Maipo)",
                    "count": 3
                }
            ],
            "mem": {
                "total_in_bytes": 24605356032,
                "free_in_bytes": 2476093440,
                "used_in_bytes": 22129262592,
                "free_percent": 10,
                "used_percent": 90
            }
        },
        "process": {
            "cpu": {
                "percent": 0
            },
            "open_file_descriptors": {
                "min": 289,
                "max": 295,
                "avg": 291
            }
        },
        "jvm": {
            "max_uptime_in_millis": 6636138488,
            "versions": [
                {
                    "version": "12.0.1",
                    "vm_name": "OpenJDK 64-Bit Server VM",
                    "vm_version": "12.0.1+12",
                    "vm_vendor": "Oracle Corporation",
                    "bundled_jdk": true,
                    "using_bundled_jdk": true,
                    "count": 3
                }
            ],
            "mem": {
                "heap_used_in_bytes": 3336896424,
                "heap_max_in_bytes": 8589934592
            },
            "threads": 142
        },
        "fs": {
            "total_in_bytes": 145261670400,
            "free_in_bytes": 63681105920,
            "available_in_bytes": 63681105920
        },
        "plugins": [],
        "network_types": {
            "transport_types": {
                "security4": 3
            },
            "http_types": {
                "security4": 3
            }
        },
        "discovery_types": {
            "zen": 3
        }
    }
}
9

Hi @Christian_Dahlqvist,
I wanted to create mapping manually, but the number of attributes is more than 100 collectively.
do you want me to try any changes.

11

Hi @MKU,

are you using G1 GC? Can you share your jvm.options so we can take a look? Alternatively, the info is also included in the output from the _nodes API.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.