# Cant ship big log from filebeat or logstash to elastic

**URL:** <https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152>\
**Category:** Logstash\
**Created:** [August 22, 2021, 5:17am UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152 "2021-08-22T05:17:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hagaluly](https://avatars.discourse-cdn.com/v4/letter/h/d78d45/32.png) [@hagaluly](https://discuss.elastic.co/u/hagaluly)\
**Post date:** [August 22, 2021, 5:17am UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152/1 "2021-08-22T05:17:05Z")

</div>

1. i have a 3 nodes cluster with 1 master and 2 data nodes each is set for 1TB
2. i have increased both -Xms24g -Xmx24g to half my ram (48GB total)
3. i than successfully upload 140mb file from Kibana to elk from the GUI after increasing it from 100mb to 1GB

when i tried to upload same file with only logstash the process was stuck and broke elastic  
my pipeline is fairly simple

input {  
file {  
path =\> "/tmp/\*\_log"  
}  
}  
output {  
elasticsearch { hosts =\> ["localhost:9200"] }  
stdout { codec =\> rubydebug }  
}

small files works great. im not able to push big files.  
log contains 1 million rows  
i set all fields in /etc/security/limits.conf to unlimited  
any ideas what im missing?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 23, 2021, 1:41am UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152/2 "2021-08-23T01:41:42Z")

</div>

Welcome to our community! 😃

> [@hagaluly](#):
>
> when i tried to upload same file with only logstash the process was stuck and broke elastic  
> my pipeline is fairly simple

Stuck how? What do the logs show? What is and isn't happening?  
We need a bit more detail to be able to help 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2021, 2:51am UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152/3 "2021-08-23T02:51:36Z")

</div>

Just to add what someone else pointed out on Stack Overflow for this question, the ability to push files from Kibana is not the same as having a file input tail a file. You might want to add start\_position and sincedb\_path (since the file input will already have recorded the lengths of the files).

---

<div class="post-metadata">

**Author:** ![hagaluly](https://avatars.discourse-cdn.com/v4/letter/h/d78d45/32.png) [@hagaluly](https://discuss.elastic.co/u/hagaluly)\
**Post date:** [August 23, 2021, 1:01pm UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152/4 "2021-08-23T13:01:15Z")

</div>

will try

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2021, 1:01pm UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152/5 "2021-09-20T13:01:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.