# Horizontal scaling doesn't improve logshash performance with kinesis input plugin **URL:** \ **Category:** Logstash\ **Created:** [March 30, 2020, 12:37pm UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674 "2020-03-30T12:37:19Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![deepak\_deore](https://avatars.discourse-cdn.com/v4/letter/d/9fc29f/32.png) [@deepak\_deore](https://discuss.elastic.co/u/deepak_deore)\ **Post date:** [March 30, 2020, 12:37pm UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/1 "2020-03-30T12:37:20Z")
as per prometheus metrics, only 1 logstash pod is processing the kinesis input, we need to vertically scale the pods in this case. ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68f044e77666195d9463e53c4ccdd99819df446f.png) can multiple logstash not work in parallel with kinesis input? ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [March 30, 2020, 7:01pm UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/2 "2020-03-30T19:01:04Z")
@deepak_deore - what is the shard count of the Kinesis data stream from which your logstash pods are consuming the messages? ---
**Author:** ![deepak\_deore](https://avatars.discourse-cdn.com/v4/letter/d/9fc29f/32.png) [@deepak\_deore](https://discuss.elastic.co/u/deepak_deore)\ **Post date:** [March 30, 2020, 8:03pm UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/3 "2020-03-30T20:03:24Z")
there is only 1 shard is it 1 shard == 1 logstash calculation? ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [March 30, 2020, 8:31pm UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/4 "2020-03-30T20:31:06Z")
Yes, that is how the Kinesis Client Library, which the Logstash Kinesis Input Plugin uses, seems to work. Please see this and give resharding (increase the shard count) an attempt. - [https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-scaling.html](https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-scaling.html) Please keep in mind that increasing the shard count has cost implications - [https://aws.amazon.com/kinesis/data-streams/pricing/](https://aws.amazon.com/kinesis/data-streams/pricing/) ---
**Author:** ![deepak\_deore](https://avatars.discourse-cdn.com/v4/letter/d/9fc29f/32.png) [@deepak\_deore](https://discuss.elastic.co/u/deepak_deore)\ **Post date:** [March 31, 2020, 11:11am UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/5 "2020-03-31T11:11:01Z")
thanks @Rahul_Kumar4, it worked btw... logstash uses `application_name` to save the state in dynamodb, do you know if we run 2 logstash with 1 kinesis shard but set different `application_name` that way both logstash will work independently on a single kinesis shard ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [April 1, 2020, 11:36am UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/6 "2020-04-01T11:36:16Z")
That would not work. Having two different `application_names` would mean two separate `dynamo_db` tables would be created to track the checkpointing both independent of each other, so unless you find a way to share the states of these two different tables between them, the two instances would not know how far have the records have been processed in that single shard. You may actually end up reprocessing the records twice. You could do with just a single Logstash instance but if you are looking for scaling, you would have to increase the shard count. These lines in that link have more context `Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard. However, one worker can process any number of shards, so it's fine if the number of shards exceeds the number of instances.` ---
**Author:** ![deepak\_deore](https://avatars.discourse-cdn.com/v4/letter/d/9fc29f/32.png) [@deepak\_deore](https://discuss.elastic.co/u/deepak_deore)\ **Post date:** [April 1, 2020, 11:42am UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/7 "2020-04-01T11:42:31Z")
thanks for the info, clear now ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 29, 2020, 11:42am UTC](https://discuss.elastic.co/t/horizontal-scaling-doesnt-improve-logshash-performance-with-kinesis-input-plugin/225674/8 "2020-04-29T11:42:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.