# Scaling Logstash
**URL:** https://discuss.elastic.co/t/scaling-logstash/90158
**Category:** Logstash
**Created:** [June 20, 2017, 8:35pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158 "2017-06-20T20:35:08Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Rory](https://discuss.elastic.co/u/Rory)
#### Post date: [June 20, 2017, 8:35pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/1 "2017-06-20T20:35:09Z")
Hi All,
We have a system where we have to rebuild an index every night which has 300 Million documents. Our source of truth is SQL Server and every night we want to build a new index. Our document size is about 2-3 KB. I was wondering if there is some documentation out there that can help me scale logstash by adding multiple instances. What is the best throughput I can achieve using logstash.
Thanks.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 21, 2017, 5:48am UTC](https://discuss.elastic.co/t/scaling-logstash/90158/2 "2017-06-21T05:48:56Z")
Since the jdbc input itself is single-threaded you'll probably want multiple jdbc inputs to parallelize the processing and avoid making the input a bottleneck. Perhaps you can shard the input data on some field? For example, if you want ten jdbc inputs the first one would use the query
```
SELECT * FROM tablename WHERE id % 10 = 0
```
the second one
```
SELECT * FROM tablename WHERE id % 10 = 1
```
and so on (assuming the id column is an integer with sufficiently random distribution of values).
---
### Author:  [@Rory](https://discuss.elastic.co/u/Rory)
#### Post date: [June 21, 2017, 1:14pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/3 "2017-06-21T13:14:02Z")
Thanks for the reply. I have attempted to do something similar, we have an id field which is an integer. Will I get a better throughput if I use one Logstash instance with multiple inputs or multiple Logstash instances with one jdbc input?
Thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 21, 2017, 1:34pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/4 "2017-06-21T13:34:00Z")
I'm not sure if it matters. The other aspects that come to play can be tweaked.
---
### Author:  [@Rory](https://discuss.elastic.co/u/Rory)
#### Post date: [June 22, 2017, 5:25pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/5 "2017-06-22T17:25:08Z")
@magnusbaeck - I'm using linux servers as my logstash instances. I have 10 such instances set up and I am dividing the data by performing a % on the id column. Each instance has 2GB RAM and 2 cores. With this set up I get a throughput of about 250 docs/second. I'm using all default settings, are there any settings that I should tweak(in logstash or in my ES Cluster) to get a better performance?
Thanks.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 22, 2017, 6:45pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/6 "2017-06-22T18:45:48Z")
250 docs/s per instance I suppose? That's slow but not unreasonably slow if you have many filters. Is the pipeline's bottleneck Logstash at all or is it ES that's limiting the throughput?
---
### Author:  [@Rory](https://discuss.elastic.co/u/Rory)
#### Post date: [June 22, 2017, 8:02pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/7 "2017-06-22T20:02:45Z")
When only one logstash instance is running, ES can ingest at 250 docs/second. With all 10 instances running I get a throughput of 2000 docs/second. I have a ruby filter in my config that deletes all attributes in the event that are null. Since with all 10 instances running the through is not 2500/second, is it safe to assume that part of the bottleneck is the Cluster? If so, how can I improve that?
Thanks.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 22, 2017, 8:09pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/8 "2017-06-22T20:09:56Z")
What is the specification of your Elasticsearch cluster? How many nodes do you have? How many indices/shards are you actively indexing into?
---
### Author:  [@Rory](https://discuss.elastic.co/u/Rory)
#### Post date: [June 22, 2017, 8:33pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/9 "2017-06-22T20:33:17Z")
I have a 4 node cluster. Each node has 64GB memory and 1.2TB disk. I will be actively ingesting into 1 index at a time with about 30 Million documents(there are a total of 10 such indices which need to be refreshed and total number of docs is about 250-350Million) and the number of shards is 32. The number of shards is a lot but with testing I have found that the query performance is ideal for this index with 32 shards(there are about 10-15 different aggregations per query).
Thanks.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 22, 2017, 8:40pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/10 "2017-06-22T20:40:30Z")
Assuming you are on a recent version of Logstash, could you please try starting it with an [internal pipeline size (-b)](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html#command-line-flags) of e.g. 1000 or 2000 to see if that makes a difference?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 20, 2017, 8:40pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158/11 "2017-07-20T20:40:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.