BulkIndexError: ('200 document(s) failed to index.'

[subham_p (Subham Panda)]

Hello Team,

I am trying to push approx 10000 data by converting the pandas data frame to elk.

But it fails to push all, some times it showing following error

failed to establish a new connection errno 111 connection refused
failed to establish a new connection errno 110 connection timeout

#converting to dataframe
df = DataFrame(result,columns=["summarized_test_data_id","package_id","panel_id"]

#formatting the data
def generator(df1):
for c, line in enumerate(df1):
yield {
'_index':''index_name',
#'_type':''index_name1',
'_type':'_doc',
'_id':line.get('summarized_test_data_id'),
'_source':{
'summarized_test_data_id':line.get('summarized_test_data_id',['No Data']),
'package_id':line.get('package_id',['No Data']),
'panel_id':line.get('panel_id',['No Data']),
'test_id':line.get('test_id',['No Data']),
'loinc_nm':line.get('loinc_nm',['No Data']),
}
}

res=helpers.bulk(es,generator(df1),refresh=True,request_timeout=3600)

it seems the following error,But in actual it pushed only 10 records to index

BulkIndexError: ('200 document(s) failed to index.', [{'index': {'_index': 'in-dev-patient-summary-research5', '_type': '_doc', '_id': '15765', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse', 'caused_by': {'type': 'json_parse_exception', 'reason': "Non-standard token 'NaN': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS to allow\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 66]"}},

[warkolm (Mark Walkom)]

Welcome to our community!

It looks like you have some non-numeric data in a numeric field?

[subham_p (Subham Panda)]

Dear warkolm,
As you said,this works fine numeric field only..
But in actual my data format is in this manner..

"in-dev-patient-summary-research6" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "created_dttm" : {
          "type" : "date"
        },
        "created_usr_id" : {
          "type" : "long"
        },
        "hos_patient_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "isactive" : {
          "type" : "boolean"
        },
        "loinc_nm" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "org_grp_id" : {
          "type" : "long"
        },
        "org_id" : {
          "type" : "long"
        },
        "panel_id" : {
          "type" : "float"
        },
        "patient_id" : {
          "type" : "long"
        },
        "summarized_test_data_id" : {
          "type" : "long"
        },
        "test_id" : {
          "type" : "float"
        },
        "test_result" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "test_type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }

"12345" : {
            "created_dttm" : "2020-04-08 17:17:56.379728",
            "created_usr_id" : 123456789,
            "hos_patient_id" : "P00000001",
            "isactive" : true,
            "loinc_nm" : null,
            "modified_dttm" : "None",
            "modified_usr_id" : null,
            "org_grp_id" : 1234,
            "org_id" : 2345,
            "package_id" : null,
            "panel_id" : null,
            "patient_id" : 1234567891011,
            "patient_visit_id" : 3456,
            "summarized_test_data_id" : 12345,
            "test_id" : 10111213,
            "test_result" : null,
            "test_type" : null,
            "version_id" : 1
          }, 

Can I store this format in elk ?

[warkolm (Mark Walkom)]

Yes you can, but you cannot create a field as numeric and then try to store non-numeric data in there.
You will need to remove/alter it before indexing.