Exception in indexing function: ElasticsearchException got in indexing function: TransportError

Elastic Stack Elasticsearch
1

Hello All,
I was ingesting the data into the ElasticCloud and suddenly I got this error before the competition of the data I was having the data of 14BG and at 13.5 GB data are being completed but this error has come and it got stopped now.

[17:15:47]{completed_searching2.py:346}ERROR- Error Occured: Exception in indexing function: ElasticsearchException got in indexing function: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [419990290/400.5mb], which is larger than the limit of [410412646/391.3mb], real usage: [419976320/400.5mb], new bytes reserved: [13970/13.6kb], usages [request=17232/16.8kb, fielddata=21250/20.7kb, in_flight_requests=13970/13.6kb, model_inference=0/0b, accounting=779820/761.5kb]')

2

How are you ingesting the data into Elasticsearch?

3

I have made a script using this script I was ingesting to the elastic cloud and before completion, I got this error and unable to find the solution can you please help me to have the solution for this error.
data was around 14BG and 13.5 GB are completed and now it is stopped.

4

What language is the script? Are you using an Elasticsearch client? Are you using _bulk?

5

I am using python language for script

6

This is important too.

7

I think no
but how will I know it is _bulk

8

Because you will be calling the bulk API end point. Which it doesn't sound like you are, which is not ideal.

9

please tell me how can I resolve it?

10

if that was the problem so why 13.5 GB of data are ingested successfully and why only remaining 500 MB data does not and why right now i am getting this issue.
never mind of my question i am very curious and love to learn so to get to know i am asking i hope i will have answer from you
Thank you

11

Basically you are sending too much data to Elasticsearch.

Using bulk reduces the overhead when talking to Elasticsearch for indexing. Which should minimise this.

12

so basically doc size is very big you mean right?
so after getting this error can I reingest again to it because it Is on production and client will need to work on it and I have very little time so tell me, sir
can I reingest and will I not lose all data?

Thank you

13

Potentially.

14

I can see that you opened the same question twice (ElasticsearchException got in indexing function: TransportError - #3 by sanjaysh) and that you are also asking me privately.

Let's keep the conversation in one single place. Let say here.

15

From the code I have seen, you are using single index request per line of CSV which is inefficient but I'm suspecting something else here. What is the HEAP size? Could you share the full elasticsearch logs please?

It could have been easier to use Logstash or filebeat by the way to import your data in elasticsearch.

16

Sorry, But I really don't know about the HEAP size.
The file has millions of rows all data pasting here is not possible. I hope you can understand. But I am giving the last few lines of the log file. I hope it will help you to understand the error. And I will hope to have the solution from you, sir...

[17:14:59]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/1902f4f3ed91842c0cd35d65d67047a2b2166fda [status:201 request:2.216s]

[17:14:59]{completed_searching2.py:307}INFO- created

[17:15:07]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/0a7041bd2674041636c6d5cfd88da4a72fd96034 [status:201 request:4.101s]

[17:15:07]{completed_searching2.py:307}INFO- created

[17:15:13]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/3acb1f3607684a44b0233c2c26b30501c0048d2b [status:201 request:2.505s]

[17:15:13]{completed_searching2.py:307}INFO- created

[17:15:20]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/09f60ebb86a57345a3eaeabab5cdb466549498f4 [status:201 request:2.701s]

[17:15:20]{completed_searching2.py:307}INFO- created

[17:15:28]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/fae1aee43334ee9b1cfc6ba58aa86429fc3b80f9 [status:201 request:4.677s]

[17:15:28]{completed_searching2.py:307}INFO- created

[17:15:40]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:121}INFO- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/5296f2495b460d8a957ab5724b3ed41d40b33003 [status:201 request:4.412s]

[17:15:40]{completed_searching2.py:307}INFO- created

[17:15:47]{/home/atul/Import-To-ES/.be/lib/python3.6/site-packages/elasticsearch/connection/base.py:149}WARNING- POST https://7a3c250340134b0fbc4d4d22d1d23be9.us-west1.gcp.cloud.es.io:9243/ccgp1/_doc/acc585c05d1f05cf89898517d04b83a4c732318a [status:429 request:0.023s]

[17:15:47]{completed_searching2.py:346}ERROR- Error Occured: Exception in indexing function: ElasticsearchException got in indexing function: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [419990290/400.5mb], which is larger than the limit of [410412646/391.3mb], real usage: [419976320/400.5mb], new bytes reserved: [13970/13.6kb], usages [request=17232/16.8kb, fielddata=21250/20.7kb, in_flight_requests=13970/13.6kb, model_inference=0/0b, accounting=779820/761.5kb]')
17

So could you share the full elasticsearch logs please as I asked previously?

18

A single line would be enough. But that's probably not the problem here.

19

But here I am not looking for the option for sharing the file. So should I copy and paste here???

20

Copy/paste and format as you did previously.
If it does not fit on the forum, upload to gist.github.com and paste the link here.