# Using pipeline - java.lang.OutOfMemoryError: Java heap space

**URL:** <https://discuss.elastic.co/t/using-pipeline-java-lang-outofmemoryerror-java-heap-space/153952>\
**Category:** Elasticsearch\
**Created:** [October 25, 2018, 8:55am UTC](https://discuss.elastic.co/t/using-pipeline-java-lang-outofmemoryerror-java-heap-space/153952 "2018-10-25T08:55:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ilma](https://avatars.discourse-cdn.com/v4/letter/i/ce7236/32.png) [@ilma](https://discuss.elastic.co/u/ilma)\
**Post date:** [October 25, 2018, 8:55am UTC](https://discuss.elastic.co/t/using-pipeline-java-lang-outofmemoryerror-java-heap-space/153952/1 "2018-10-25T08:55:54Z")

</div>

using pipeline ngest-attachment logstash로 elasticsearch로 output할때 생긴문제입니다.

파이프 라인은 아래와같습니다. -------------

PUT \_ingest/pipeline/attachment  
{  
"description" : "Extract attachment information",

"processors" : [  
{  
"attachment" : {  
"field" : "data",  
"indexed\_chars" : "-1"  
},  
"remove": {  
"field": "data"  
}  
}  
]  
}

* * *

\*\* input 데이터의 크기는 300row쯤되고 파일의 용량은 200M ~ 500M 정도됩니다.

filename필드에 경로가 쓰여져 있습니다. ex) /home/search/file/1/145/1948-efg-213.pdf

rubyfile.txt 를 conf로 지정해서 실행시에

java.lang.OutOfMemoryError: Java heap space  
Dumping heap to java\_pid27866.hprof ...  
Heap dump file created [4237042841 bytes in 23.982 secs]  
[2018-10-22T10:05:56,177][ERROR][logstash.filters.ruby] Ruby exception occurred: Java heap space  
[2018-10-22T10:05:57,743][ERROR][logstash.filters.ruby] Ruby exception occurred: Java heap space

이러한 에러가 발생합니다.

데이터가 너무큰가해서

select \* From tbl\_file where sitename = '국내' and sourcename = 'Test RA-ebook' limit 1 ( 하나의 row만 가져옴)

이러한 쿼리로 실행하였을때에는

[2018-10-22T10:08:36,631][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=\>"exception", "reason"=\>"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character a", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"java.lang.IllegalArgumentException: Illegal base64 character a", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Illegal base64 character a"}}, "header"=\>{"processor\_type"=\>"attachment"}})

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2018, 9:02am UTC](https://discuss.elastic.co/t/using-pipeline-java-lang-outofmemoryerror-java-heap-space/153952/2 "2018-11-22T09:02:31Z")

</div>

Closed on November 22, 2018, 9:02am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.