# Process 200 events per second within 100ms **URL:** \ **Category:** Logstash\ **Created:** [April 28, 2019, 8:21pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816 "2019-04-28T20:21:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![rahulkothanath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulkothanath/32/99122_2.png) [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Post date:** [April 28, 2019, 8:21pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/1 "2019-04-28T20:21:56Z")
I have a use case in which I have to process 200 events/second and each event should not take more than 100 ms to process. I have used NIFI for this and it is taking around 500 ms for an event to complete an I am planning to use logstash now. details: Input: Kafka topic output :Json Data is of fixed length text (2000) an I have to create 30 json fields out of it using filters. is logstash right choice for this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 29, 2019, 4:19am UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/2 "2019-04-29T04:19:27Z")
What drives this requirement? How are you measuring latency? Where are you outputting the generated JSOn to? ---
**Author:** ![ariemenschneider](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ariemenschneider/32/33564_2.png) [@ariemenschneider](https://discuss.elastic.co/u/ariemenschneider)\ **Post date:** [May 6, 2019, 12:09pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/3 "2019-05-06T12:09:45Z")
Is the 100ms latency a "hard" requirement, i.e. each event **must** be completed in less than 100ms or is it a "soft" requirement, i.e. each event should be processed in average in a time less than 100ms? As Logstash is a Java application, there may be situations (during GC), where an event could take longer than 100ms to be processed. This can be minimized with careful JVM sizing, but I doubt it could be completely avoided. From a performance, perspective, Logstash can handle this requirements with ease (screenshot taken of 2 parallel LS instances, each with 6 vCPUs and 6GB Heap space handling ~20 Input pipelines, some of them quite complex: ![Screenshot_2019-05-06%20Monitoring%20-%20Logstash](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d34fc8139312ef27db4f1ae7c8b768c5b72445b9.png) ---
**Author:** ![rahulkothanath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulkothanath/32/99122_2.png) [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Post date:** [May 6, 2019, 1:07pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/4 "2019-05-06T13:07:30Z")
The requirement is like below: customer makes a transaction and I receive corresponding data in real time. I have to parse the received data based on grok pattern and push to kafka as json. another program will make some business rules and give reply back to customer . I have to convert received customer data from kafka using log stash filter into json and push to kafka within 100ms. ---
**Author:** ![rahulkothanath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahulkothanath/32/99122_2.png) [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Post date:** [May 6, 2019, 1:10pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/5 "2019-05-06T13:10:39Z")
100 ms is a soft requirement and in worst case I can take upto 200ms. I have 3 machines each with 32GB RAM and 8 cores to accomplish this. I do have some complex filtering logic happening in logstash which makes me doubtful about achieving this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 3, 2019, 1:10pm UTC](https://discuss.elastic.co/t/process-200-events-per-second-within-100ms/178816/6 "2019-06-03T13:10:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.