# Logsatsh not stashing a huge file

**URL:** <https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530>\
**Category:** Logstash\
**Created:** [April 5, 2019, 7:16am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530 "2019-04-05T07:16:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sana1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1/32/46439_2.png) [@sana1](https://discuss.elastic.co/u/sana1)\
**Post date:** [April 5, 2019, 7:16am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/1 "2019-04-05T07:16:55Z")

</div>

I have a file of ~ 23gb, I am using filebeat for the logstash input, but Logstash only parses a few lines from the start of the log file, what should I do?

---

<div class="post-metadata">

**Author:** ![elastock](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elastock/32/35672_2.png) [@elastock](https://discuss.elastic.co/u/elastock)\
**Post date:** [April 5, 2019, 7:36am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/2 "2019-04-05T07:36:10Z")

</div>

1 - Say hello  
2 - Explain a bit more your problem ..  
3 - Show your configs file ..

It's a good start. 🙂

---

<div class="post-metadata">

**Author:** ![sana1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1/32/46439_2.png) [@sana1](https://discuss.elastic.co/u/sana1)\
**Post date:** [April 5, 2019, 7:40am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/3 "2019-04-05T07:40:33Z")

</div>

Hello I am Sana From NUST.  
Newbie to ELK Stack.

I want to parse a data of ~23 GB file. For few lines of code my Code worked fine. But for 23 gb File, it got stucked in between.

what are the checks for logstash to deal with such a huge file.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 5, 2019, 7:44am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/4 "2019-04-05T07:44:01Z")

</div>

What does your config look like? What does data in the file look like?

---

<div class="post-metadata">

**Author:** ![sana1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1/32/46439_2.png) [@sana1](https://discuss.elastic.co/u/sana1)\
**Post date:** [April 5, 2019, 8:02am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/5 "2019-04-05T08:02:52Z")

</div>

log lines : 90-50-31: ill ill you are You  
log lines: 90-50-31: ill till you are You  
log lines : 90-50-31: ill end you are You

input {  
beats {  
port =\> 5044  
}  
}  
if [ty] =~ "YOU" or "you" {  
mutate {  
gsub =\> ["field", "[a-zA-Z]+" , "" ]  
}  
} else {  
mutate { gsub =\> ["field", "[0-9]+" , "" ] }  
}

output {  
elasticsearch {  
hosts =\> "[http://localhost:9200](http://localhost:9200)"

```
}
stdout {
codec => rubydebug

```

}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 5, 2019, 8:09am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/6 "2019-04-05T08:09:48Z")

</div>

Do you have 23GB of log lines like that? What does your beats config look like?

---

<div class="post-metadata">

**Author:** ![sana1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1/32/46439_2.png) [@sana1](https://discuss.elastic.co/u/sana1)\
**Post date:** [April 5, 2019, 9:01am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/7 "2019-04-05T09:01:48Z")

</div>

yes logs have almost the same format,

beats  
filebeat.prospectors:

- type: log  
enabled: true  
paths:
  - /home/matrix/mylogs.log  
output.logstash:  
hosts: ["localhost:5044"]

---

<div class="post-metadata">

**Author:** ![sana1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sana1/32/46439_2.png) [@sana1](https://discuss.elastic.co/u/sana1)\
**Post date:** [April 5, 2019, 10:22am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/8 "2019-04-05T10:22:43Z")

</div>

This question can be rephrased as how to deal with huge log files using ELK stack?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2019, 10:22am UTC](https://discuss.elastic.co/t/logsatsh-not-stashing-a-huge-file/175530/9 "2019-05-03T10:22:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.