# Consume a large of data in a file with Logstash
**URL:** https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322
**Category:** Logstash
**Created:** [November 11, 2015, 11:59am UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322 "2015-11-11T11:59:13Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [November 11, 2015, 11:59am UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/1 "2015-11-11T11:59:14Z")
Hi,
I have a logstash that have consume nginx logs and another applications in multiples input file in logstash configuration, but when we increase the acess in the website, we have a lot of access and the logstash delay to collect the information and we have a role in dashboard.
My redis queue still clear all the time.
Someone has a tip about this?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 11, 2015, 12:03pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/2 "2015-11-11T12:03:23Z")
How many messages is Logstash processing per second? Do you have any time consuming filters? How's the CPU load?
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [November 11, 2015, 1:51pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/3 "2015-11-11T13:51:38Z")
In my index have 109090727 documents. So we have 2525 by second.
I have many filters with several business rules in logstash what read the logs files.
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [November 11, 2015, 1:52pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/4 "2015-11-11T13:52:34Z")
the problem happens when we have a lot of access in nginx ☹
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [November 11, 2015, 2:26pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/5 "2015-11-11T14:26:46Z")
Another information is I have only one logstash to consume several logs not only the nginx log
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 11, 2015, 2:42pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/6 "2015-11-11T14:42:01Z")
Well, it sounds like you might need to profile and optimize your filters a bit. If you show us what you've got we might be able to help out. To increase throughput make sure you saturate your CPUs, e.g. by increasing the number of filter workers with the `-w` startup option.
---
### Author:  [@Jeferson\_Martins](https://discuss.elastic.co/u/Jeferson_Martins)
#### Post date: [November 11, 2015, 4:37pm UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/7 "2015-11-11T16:37:40Z")
We talk about and we will make some changes in Logstash Log Reader and in the Logstash Indexer.
😄
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:23am UTC](https://discuss.elastic.co/t/consume-a-large-of-data-in-a-file-with-logstash/34322/8 "2017-07-06T05:23:08Z")