TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large,

Elastic Stack Elasticsearch
1

elasticsearch
Version: 7.8.0
Summary: Python client for Elasticsearch

TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [2128807074/1.9gb], which is larger than the limit of [2040109465/1.8gb], real usage: [2128646072/1.9gb], new bytes reserved: [161002/157.2kb], usages [request=0/0b, fielddata=1186/1.1kb, in_flight_requests=161002/157.2kb, model_inference=0/0b, accounting=10784126/10.2mb]')

Is there any way to fix this error?

2

Could you tell what code is causing this?

3

Dear dadoonet,
I am trying to push more than 10000 data to elk.

df1=df.to_dict('records')
def generator(df1):
    for c, line in enumerate(df1):
        yield {
        '_index':'in-dev-patient-summary-research_str', 
        '_id':line.get('summarized_test_data_id'),
        '_source':{
            'loinc_nm':line.get('loinc_nm'),
            'test_type':line.get('test_type'),
            'hos_patient_id':line.get('hos_patient_id'),
            'isactive':line.get('isactive'),
            
        }
             }
res=helpers.bulk(es,generator(df1),request_timeout=3600)
4

Split what you have to send to elasticsearch in smaller bulk requests.

I'm normally not sending more than 10000 requests at once but it depends on the size of the documents.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.