# Out of memory error and duplicate rows **URL:** \ **Category:** Logstash\ **Created:** [December 23, 2022, 3:46pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932 "2022-12-23T15:46:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![darius12](https://avatars.discourse-cdn.com/v4/letter/d/76d3ee/32.png) [@darius12](https://discuss.elastic.co/u/darius12)\ **Post date:** [December 23, 2022, 3:46pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/1 "2022-12-23T15:46:20Z")
getting this error when trying to index using logstash: warning: thread "[main]\>worker1" terminated with exception (report\_on\_exception is true): java.lang.OutOfMemoryError: UTF16 String size is 1371255266, should be less than 1073741823 also, the customer\_id is unique in the customer table, however the logstash is repeating the records more than 500 times per same customer\_id ======== Heap size is 32 GB, server has 64 GB memory We are attempting to index 1 terabyte of data, which is about 500,000 of our records. Our logstash script is below. Any help with the java.lang.OutofMemoryError and repeating rows would be greatly appreciated. ```auto input { jdbc { jdbc_connection_string => "jdbc:sqlserver://SERVERA-DB:1433;databaseName=db_1;integratedsecurity=true;" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_user => "bob_hanson" jdbc_paging_enabled => "true" jdbc_page_size => "5000" statement => " SELECT customer_id, first_name, middle_initial, last_name, address1, address2, city, zip_code from [customers] tags => ["customer_search_tags"] } } filter { jdbc_streaming { jdbc_connection_string => "jdbc:sqlserver://SERVERA-DB:1433;databaseName=db_1;integratedsecurity=true;" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_user => "bob_hanson" statement => " SELECT document_id, document, document_content_type, document_name FROM Customer_Documents cd WHERE document is not null AND document_content_type in ('application/pdf', 'application/msword', 'text/plain','application/atom+xml','application/msaccess','application/msexcel','application/vnd.ms-excel','application/vnd.ms-officetheme','application/vnd.ms-outlook', 'application/vnd.ms-powerpoint','application/vnd.ms-word.document.macroEnabled.12','application/vnd.openxmlformats-officedocument.pres','application/vnd.openxmlformats-officedocument.spre','application/vnd.openxmlformats-officedocument.word','application/vnd.visio','application/x-cpy', 'message/rfc822','text/css','text/html', 'text/xml') AND cd.customer_id = :cc_customer_id" parameters => { "cc_customer_id" => "customer_id"} target => "attachments" } } output { elasticsearch { hosts => [https://XXXXXXXXXXXX:9200] cacert => "E:\logstash\config\blog_cert.pem" ssl => true ssl_certificate_verification => false pipeline => "attachments" index => "customer_search_entries" user => "elastic" password => "XXXXXXXX==" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 23, 2022, 6:21pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/2 "2022-12-23T18:21:22Z")
> [@darius12](#): > > java.lang.OutOfMemoryError: UTF16 String size is 1371255266, should be less than 1073741823 Is it possible that one of your documents is over 1 GB? A few years ago the maximum string size in Java was [reduced](https://bugs.openjdk.org/browse/JDK-8190429) from 2 GB (because each UTF-16 character uses 2 bytes and it uses a 32-bit string length) to 1 GB (a side-effect of compact strings which are byte arrays rather than char arrays). Is the pipeline getting restarted over and over again due to this exception? If so, perhaps use the customer\_id as the document\_id in elasticsearch, so that it will just keep overwriting the document. ---
**Author:** ![darius12](https://avatars.discourse-cdn.com/v4/letter/d/76d3ee/32.png) [@darius12](https://discuss.elastic.co/u/darius12)\ **Post date:** [December 23, 2022, 6:46pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/3 "2022-12-23T18:46:32Z")
Yes, definitely there are some documents over 1 GB in size. Is there a way to index in this case? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 23, 2022, 7:25pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/4 "2022-12-23T19:25:02Z")
I'm not sure, but I doubt it. Even if you could cast it to some other type that logstash could process I suspect it would then get the same issue on the elasticsearch side. The default limit on document size in elasticsearch is 100 MB, Lucene supports up to 2 GB but elastic [recommend against it](https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html#maximum-document-size). ---
**Author:** ![darius12](https://avatars.discourse-cdn.com/v4/letter/d/76d3ee/32.png) [@darius12](https://discuss.elastic.co/u/darius12)\ **Post date:** [December 27, 2022, 2:29pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/5 "2022-12-27T14:29:06Z")
thanks for the helpful feedback! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 24, 2023, 2:29pm UTC](https://discuss.elastic.co/t/out-of-memory-error-and-duplicate-rows/321932/6 "2023-01-24T14:29:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.