Using pipeline - java.lang.OutOfMemoryError: Java heap space

Elastic Stack Elasticsearch
1

using pipeline ngest-attachment logstash로 elasticsearch로 output할때 생긴문제입니다.

파이프 라인은 아래와같습니다. -------------

PUT _ingest/pipeline/attachment
{
"description" : "Extract attachment information",

"processors" : [
{
"attachment" : {
"field" : "data",
"indexed_chars" : "-1"
},
"remove": {
"field": "data"
}
}
]
}

** input 데이터의 크기는 300row쯤되고 파일의 용량은 200M ~ 500M 정도됩니다.

filename필드에 경로가 쓰여져 있습니다. ex) /home/search/file/1/145/1948-efg-213.pdf

rubyfile.txt 를 conf로 지정해서 실행시에

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid27866.hprof ...
Heap dump file created [4237042841 bytes in 23.982 secs]
[2018-10-22T10:05:56,177][ERROR][logstash.filters.ruby ] Ruby exception occurred: Java heap space
[2018-10-22T10:05:57,743][ERROR][logstash.filters.ruby ] Ruby exception occurred: Java heap space

이러한 에러가 발생합니다.

데이터가 너무큰가해서

select * From tbl_file where sitename = '국내' and sourcename = 'Test RA-ebook' limit 1 ( 하나의 row만 가져옴)

이러한 쿼리로 실행하였을때에는

[2018-10-22T10:08:36,631][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"exception", "reason"=>"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Illegal base64 character a", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"java.lang.IllegalArgumentException: Illegal base64 character a", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Illegal base64 character a"}}, "header"=>{"processor_type"=>"attachment"}})

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.