# Slow processing in Logstash with S3 input

**URL:** <https://discuss.elastic.co/t/slow-processing-in-logstash-with-s3-input/33940>\
**Category:** Logstash\
**Created:** [November 5, 2015, 11:48pm UTC](https://discuss.elastic.co/t/slow-processing-in-logstash-with-s3-input/33940 "2015-11-05T23:48:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrey](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@andrey](https://discuss.elastic.co/u/andrey)\
**Post date:** [November 5, 2015, 11:48pm UTC](https://discuss.elastic.co/t/slow-processing-in-logstash-with-s3-input/33940/1 "2015-11-05T23:48:50Z")

</div>

Hi,

We are using Logstash 2.0 to move a couple of months of log data (3000 files, 10G total) from Amazon S3 into Elasticsearch (1.7.1), but it's taking a very long time - roughly 1 hour to get through a day of data (200-300M, 1M lines). Only a small part of the incoming data needs to be indexed into ES (~100 docs per day), while the rest is discarded by our LS filters.

Logstash is configured with an S3 input, various filters to transform entries into JSON, and an Elasticsearch output:

> input { s3 { ... bucket =\> "${s3Bucket}" prefix =\> "${s3Prefix}" delete =\> true backup\_to\_bucket =\> "${s3Bucket}" backup\_add\_prefix =\> "archive/" ...
> 
> filter {  
> grok {  
> match =\> {"message" =\> "%{IP:client}\t%{NUMBER:[@metadata][timestamp]}\t%{QS:[@metadata][payload]}\t%{QS:agent}\t%{QS:[@metadata][referrer]}"}  
> ... plus several others ...
> 
> output {  
> if "\_noresults" in [tags] {  
> # ignore event  
> } else if "\_jsonparsefailure" in [tags] or "\_grokparsefailure" in [tags] {  
> # log parse error  
> stdout {  
> codec =\> rubydebug  
> }  
> } else if [tags] {  
> # log unknown tag(s)  
> stdout {  
> codec =\> rubydebug  
> }  
> } else if [@metadata][type] {  
> elasticsearch {  
> hosts =\> ["${esHost}:9200"]  
> action =\> "update"  
> doc\_as\_upsert =\> true  
> index =\> "analytics-%{[@metadata][type]}-%{+YYYY.MM.dd}"  
> document\_id =\> "%{[@metadata][doc\_id]}"  
> document\_type =\> "impression"  
> template =\> "/etc/logstash/analytics-template-v0.1.json"  
> template\_name =\> "analytics"  
> template\_overwrite =\> true  
> }  
> } else {  
> # log uncaught error  
> stdout {  
> codec =\> rubydebug  
> }  
> }

Our S3 files consist of lines like below, except some will have JSON in the third field (those are the ones we index):  
`11.222.333.444	1446649267.395	"-"	"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"	"http://referrer"`

Since so little data ends up indexed into ES, I would assume the problem lies in the input or filters?

Are there any performance settings we should set? With default settings, CPU ranges from 20-300% and heap seems OK (eden 10M/32M, old 63M/114M).

Would appreciate any hints about how to debug and what settings we should play with.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:23am UTC](https://discuss.elastic.co/t/slow-processing-in-logstash-with-s3-input/33940/2 "2017-07-06T05:23:42Z")

</div>