How to read textfiles dynamically from folder and keep listining without jump out Heap Size Memory

(samyo) #1

Hello folks,
i am using Elasticsearch and Logstash Version 6.3.1 , and Java version 8 , on Ubunto Virtualmachine , with 8GB RAM and 50GB for the Memory .
I am trying to read and match 8000 textfiles from folder using multline Codec after filter them using Logstash , and send them to Elasticsearch and keep listining from folder if any new textfile has been added so logstash will read it and send it to elastic and so on.

My Input seems not to be Confidence , cause after matching and sending about 5000 textfiles , logstash jump out of Java Heap Size Memory Space.
Can anybody please correct my Input bellow:

input {
file {
path => "/home/sam/Desktop/folder2/*.txt"
sincedb_path => "/dev/null"
start_position => "beginning"
stat_interval => 300
discover_interval => 1
ignore_older => 864000
close_older => 20
max_open_files => 102400
        codec => multiline
       {
            pattern => "^\s"
            negate => true
            what => previous		
        }
}
}

how to write an Input that listining from folder if any textfiles has been added , with out jumping out of Heap Size Memory Space????

i will be thankfull for any Idea.