# Logstash input (kafka) add\_field very slow

**URL:** <https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541>\
**Category:** Logstash\
**Created:** [September 2, 2015, 4:23pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541 "2015-09-02T16:23:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![L8apex32](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@L8apex32](https://discuss.elastic.co/u/L8apex32)\
**Post date:** [September 2, 2015, 4:23pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/1 "2015-09-02T16:23:06Z")

</div>

We are working towards very fast ingest into elastic search and found an unexpected slow spot. We have LS pulling from kafka which it can do very fast (5 consumer threads) - over 50k eps. However, if we add a single "add\_field" do the kafka input, the events per second drops by 24k eps. This seems like a problem that could be addressed.

This testing is done with no filter statements and output to null. If we leave the add\_field out of the kafka input statement, we can add lots of filter conditionals and complex grok statements (parsing up to 16 fields per message) and still outperform the performance of adding the single add\_field statement to the input filter.

If we use "add\_field" in the filter section, it does not slow LS down hardly at all.

Is there a way we could have the performance hit of "add\_field" looked into when used in the input section?

Thanks.

---

<div class="post-metadata">

**Author:** ![Johntdyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johntdyer/32/3424_2.png) [@Johntdyer](https://discuss.elastic.co/u/Johntdyer)\
**Post date:** [February 25, 2016, 5:00am UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/2 "2016-02-25T05:00:55Z")

</div>

Just out of curiously what where the specs of the system you were benchmarking with ?

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [February 25, 2016, 1:29pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/3 "2016-02-25T13:29:30Z")

</div>

Does the add field do any %blah token replacement? I've seen that being a bottleneck preciously.

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [February 25, 2016, 1:31pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/4 "2016-02-25T13:31:06Z")

</div>

Ouch, this is an old thread.

---

<div class="post-metadata">

**Author:** ![L8apex32](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@L8apex32](https://discuss.elastic.co/u/L8apex32)\
**Post date:** [February 25, 2016, 10:31pm UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/5 "2016-02-25T22:31:00Z")

</div>

It is with a modern Linux box, 16 cpus, plenty of memory.

But, he is right, it was a long time ago and I don't know if this is still  
the case of not. That was with LS 1.x and 2.x is current.  
Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:09am UTC](https://discuss.elastic.co/t/logstash-input-kafka-add-field-very-slow/28541/6 "2017-07-06T05:09:43Z")

</div>


