Slow startup and no data transmit on recursive directory

[ali_tavakoli]

Hi guys

I configured my logstash to input plugin from file and send to elasticsearch. my file directory is recursive and number of file is about 500k in recursive directories and each file is average 2 MB.

logstash startup is very slow and after start it dose not send any data to Elasticsearch.

where i made mistake?

config file:

input {
    file {
        path => "/home/test/data/main/**/**/**/**/*"
        start_position => "beginning"
        codec => "json"
        sincedb_path => "/opt/elastic/.sincedb_tg_comp_1"
        close_older => 10
    }

}

output {
        elasticsearch {
                hosts => ["192.168.20.110:9200"]
                index => "test_logstash"
        }

}

in path section for input plugin i used /home/test/data/main/**/* too. but no diffrence

[magnusbaeck (Magnus Bäck)]

You're most likely hitting some kind of limit for the number of monitored files. Have you looked in the Logstash log for clues?

[ali_tavakoli]

Thanks for your reply

Yes there is no error or any kind of message there! logstash service running normally and in some case start to send data to elasticsearch but very slow! like 2 insert per second. in most time it dose not send any thing.
i raise up ubuntu open file and logstash service configuration about LimitNofile and everything but nothing happened!