# Logstash improve time performance

**URL:** <https://discuss.elastic.co/t/logstash-improve-time-performance/123741>\
**Category:** Logstash\
**Created:** [March 13, 2018, 1:45pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741 "2018-03-13T13:45:25Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 13, 2018, 1:45pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/1 "2018-03-13T13:45:25Z")

</div>

hi all,  
i have csv file that contains 4 Million lines, each line is more than 100 columns,  
i process about 30 columns and add another 10 columns, so that in total i upload about 40 columns to the kibana, in average i need 14 minutes to finish the upload,  
in order to improve the time performance i tried to manipulate

1. pipeline.workers
2. pipeline.batch.size/batch.delay:
3. JVM Xmx and Xms  
but the upload time only increased

any ideas.  
thanks in advance

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 13, 2018, 1:50pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/2 "2018-03-13T13:50:09Z")

</div>

Where are you sending the data? Are you sure Logstash is limiting performance and not the destination system? If so, how did you establish this?

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 13, 2018, 1:52pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/3 "2018-03-13T13:52:49Z")

</div>

logstash/elasticsearch/kibana are installed on the same linux machine  
the network speed is around 20Mbps

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 13, 2018, 1:58pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/4 "2018-03-13T13:58:57Z")

</div>

What does CPU usage, disk I/O and iowait look like while indexing?

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 13, 2018, 2:10pm UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/5 "2018-03-13T14:10:51Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/2/4264ad788063e4a283e0dd8ff63c2847d6049a8d.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e6371a7b009543107aa7a5640c55f88054bf940.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3bd39ff0d747f986faa498a033e4f8db56e2c1f2.png)

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 14, 2018, 8:30am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/6 "2018-03-14T08:30:32Z")

</div>

@magnusbaeck  
@paz

any idea?  
thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 14, 2018, 11:20am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/7 "2018-03-14T11:20:33Z")

</div>

What does your Logstash pipeline look like?

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 14, 2018, 11:35am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/8 "2018-03-14T11:35:29Z")

</div>

input {  
file {  
path =\> "/var/log/file"  
start\_position =\> "beginning"  
}  
}  
filter {  
csv {  
separator =\> " "  
columns =\> ["NODE\_ID", "MSISDN", "TRANSACTION\_URL", "RESPONSE\_CODE", "START\_TIME", "END\_TIME", "RESPONSE\_ORIG\_SIZE",  
"DOWNLOAD\_DATA\_SIZE","SRC\_IP", "ORIG\_DST\_IP", "USED\_DST\_IP", "SGSN\_IP", "ACC\_SESSION\_ID", "IMEI2", "USER\_AGENT",  
"COMPRESSION\_LVL", "CONTENT\_TYPE", "UPSTREAM\_SIZE","DOWNSTREAM\_SIZE", "OPTIMIZATION\_FLAG", "CA\_FLAG", "CC\_FLAG",  
"CC\_BLOCKED", "CC\_REASON", "CC\_CATEGORY", "INITIAL\_BEARER", "NETWORK\_BEARER","APN\_IP1", "POC\_HIT\_FLAG", "IMSI",  
"ORIGINAL\_REQUEST\_HOST", "ORG\_CONTENT\_LENGTH", "CONTENT\_LENGTH", "MD5\_STRING", "ORG\_BITRATE", "BITRATE",  
"IS\_BUFFER\_LIMITING", "IS\_FROM\_CACHE", "IS\_SEEK\_REQUEST", "AUDIO\_CODEC", "VIDEO\_CODEC", "MEDIA\_CONTAINER",  
"CACHED\_EXTRA\_BYTES\_LENGTH", "PARENT\_MSISDN","SEND\_NOTIFY", "IS\_CACHED", "IS\_TO\_CACHE", "TRANSCODING\_LEVEL",  
"HTTP\_METHOD", "NETWORK\_RATE", "ORG\_AUDIO\_CODEC", "ORG\_VIDEO\_CODEC","CONTENT\_ENCODING\_HEADER", "TRANSFER\_ENCODING\_HEADER",  
"MODIFIED\_CONTENT\_TYPE", "FULL\_RESOURCE\_CONTENT\_LENGTH", "RANGE\_REQUEST\_HEADER", "CACHE\_HITS","TOOLBAR\_FLAG",  
"HPI\_CACHE\_PUT\_TIME", "HITS\_TM\_GET", "TM\_IN\_TIME", "TM\_OUT\_TIME", "MEDIA\_HANDLING\_TYPE", "MEDIA\_QUALITY\_LEVEL",  
"VIDEO\_START\_TIME","NUMBER\_OF\_STALLS", "AVERAGE\_STALLS\_LENGTH", "NUMBER\_OF\_DRA\_DECREASES", "HPI\_INSTANCE", "UIDH",  
"WEB\_RESPONSE\_CODE", "WEB\_ROUND\_TRIP\_TIME","CLIENT\_ROUND\_TRIP\_TIME", "HTTP\_REQUEST\_VERSION", "HTTP\_RESPONSE\_VERSION",  
"DNS\_RESPONSE\_TIME", "DOWNLOAD\_ORIG\_SIZE", "IS\_FROM\_TUNNEL","TOOLBAR\_REQUEST\_TYPE", "REQUEST\_MODIFICATION\_INDICATION",  
"ORIGINAL\_TARGET\_URL", "PROFILE\_TEMPLATE", "ADDITIONAL\_INFO", "CLIENT\_CONNECTION\_TTL","ICL\_REASON", "MLT\_LEVEL\_REASON",  
"OT\_LEVEL\_REASON", "DRA\_LEVEL\_REASON", "ONLINE\_DRA\_LEVEL\_REASON", "CELL\_ID", "RESOLUTION\_HEIGHT", "RESOLUTION\_WIDTH",  
"PACING\_HANDLER", "PACING\_FACTOR", "CONGESTION\_LEVEL", "CONNECTION\_BW", "CONNECTION\_RTT", "IS\_NAT", "OPTIMIZATION\_STATUS",  
"IS\_CELL\_CONGESTED","SESSION\_DURATION\_REMAINDER", "COMPLETED\_DOWNLOAD", "ORIG\_DST\_PORT", "ORIG\_SRC\_PORT", "FIN\_DURATION",  
"ABM\_INFO", "CIRCLE\_MAPPING", "BACKHAUL\_RTT"] #BACKHAUL\_RTT is the 109 column  
}  
}  
filter{  
mutate{  
convert =\> {  
"DOWNSTREAM\_SIZE" =\> "integer"  
"RESPONSE\_ORIG\_SIZE" =\> "integer"  
"UPSTREAM\_SIZE" =\> "integer"  
"SESSION\_DURATION\_REMAINDER" =\> "integer"  
}  
}  
if ![DOWNSTREAM\_SIZE] or ![RESPONSE\_ORIG\_SIZE] or ![UPSTREAM\_SIZE] or ![SESSION\_DURATION\_REMAINDER]{  
drop { }  
}else if [DOWNSTREAM\_SIZE] \< 0 or [RESPONSE\_ORIG\_SIZE] \< 0 or [UPSTREAM\_SIZE]\<0{  
drop { }  
}else if [RESPONSE\_ORIG\_SIZE] \< [DOWNSTREAM\_SIZE] {  
ruby {  
code =\> "event.set('RESPONSE\_ORIG\_SIZE', event.get('DOWNSTREAM\_SIZE'))"  
}  
}  
}  
filter{  
ruby {  
code =\> "  
wanted\_fields = ['NODE\_ID', 'MSISDN', 'TRANSACTION\_URL', 'RESPONSE\_CODE', 'START\_TIME', 'END\_TIME', 'RESPONSE\_ORIG\_SIZE',  
'SRC\_IP','USER\_AGENT','CONTENT\_TYPE','UPSTREAM\_SIZE','DOWNSTREAM\_SIZE','CC\_CATEGORY','NETWORK\_BEARER',  
'POC\_HIT\_FLAG','BITRATE','IS\_FROM\_CACHE','VIDEO\_CODEC','MEDIA\_CONTAINER','MEDIA\_HANDLING\_TYPE',  
'VIDEO\_START\_TIME','NUMBER\_OF\_STALLS','AVERAGE\_STALLS\_LENGTH','CLIENT\_ROUND\_TRIP\_TIME','PROFILE\_TEMPLATE',  
'ADDITIONAL\_INFO','CELL\_ID','RESOLUTION\_HEIGHT','RESOLUTION\_WIDTH','CONGESTION\_LEVEL','CONNECTION\_RTT',  
'SESSION\_DURATION\_REMAINDER','ABM\_INFO','BACKHAUL\_RTT','@timestamp','tags']  
event.to\_hash.keys.each { |k|  
event.remove(k) unless wanted\_fields.include? k  
}  
"  
}  
}  
filter{  
mutate{  
convert =\> {  
"POC\_HIT\_FLAG" =\> "integer"  
"IS\_FROM\_CACHE" =\> "integer"  
"VIDEO\_START\_TIME" =\> "integer"  
"BITRATE" =\> "integer"  
"CLIENT\_ROUND\_TRIP\_TIME" =\> "integer"  
"CONNECTION\_RTT" =\> "integer"  
}  
}  
if ![POC\_HIT\_FLAG] or ![IS\_FROM\_CACHE]{  
ruby {  
code =\> "event.set('IS\_FROM\_CACHE',0)"  
}  
}  
if [IS\_FROM\_CACHE] \> 0 and ([POC\_HIT\_FLAG] == 2 or [POC\_HIT\_FLAG] == 3) {  
ruby {  
code =\> "event.set('IS\_FROM\_CACHE',1)"  
}  
}else{  
ruby {  
code =\> "event.set('IS\_FROM\_CACHE',0)"  
}  
}  
if ![VIDEO\_START\_TIME]{  
ruby {  
code =\> "event.set('VIDEO\_START\_TIME',0)"  
}  
}  
if [VIDEO\_START\_TIME]\<0 or [VIDEO\_START\_TIME] \> 10000 {  
ruby {  
code =\> "event.set('VIDEO\_START\_TIME',0)"  
}  
}  
if ![BITRATE]{  
ruby {  
code =\> "event.set('BITRATE',0)"  
}  
}  
if [BITRATE]\<0 {  
ruby {  
code =\> "event.set('BITRATE',0)"  
}  
}  
if !([SESSION\_DURATION\_REMAINDER]){  
ruby {  
code =\> "event.set('SESSION\_DURATION\_REMAINDER',0)"  
}  
}  
if ![CLIENT\_ROUND\_TRIP\_TIME] or ![CONNECTION\_RTT]{  
ruby {  
code =\> "event.set('TTFB',0)"  
}  
}else if [CLIENT\_ROUND\_TRIP\_TIME]\<=0 or [CONNECTION\_RTT]\<=0 or [CLIENT\_ROUND\_TRIP\_TIME] \>= 5000 or [CONNECTION\_RTT] \>= 5000 {  
ruby {  
code =\> "event.set('TTFB',0)"  
}  
}else{  
ruby {  
code =\> "(event.get('CLIENT\_ROUND\_TRIP\_TIME') + event.get('CONNECTION\_RTT')) \< 5000 ? event.set('TTFB', event.get('CLIENT\_ROUND\_TRIP\_TIME') + event.get('CONNECTION\_RTT')) : 0"  
}  
}  
mutate {  
remove\_field =\> ["POC\_HIT\_FLAG","CLIENT\_ROUND\_TRIP\_TIME","CONNECTION\_RTT"]  
}  
}  
filter {  
grok {  
match =\> {  
"BACKHAUL\_RTT" =\> [  
"%{NUMBER:BACKHAUL\_RTT\_MIN:float}%{NOTSPACE}%{NUMBER:BACKHAUL\_RTT\_AVG:float}%{NOTSPACE}%{NUMBER:BACKHAUL\_RTT\_MAX:float}"  
]  
}  
}  
mutate{  
convert =\> {  
"BACKHAUL\_RTT\_MIN" =\> "integer"  
"BACKHAUL\_RTT\_AVG" =\> "integer"  
"BACKHAUL\_RTT\_MAX" =\> "integer"  
}  
}  
if ![BACKHAUL\_RTT\_MIN] or ![BACKHAUL\_RTT\_AVG] or ![BACKHAUL\_RTT\_MAX]{  
ruby {  
code =\> "event.set('BACKHAUL\_RTT\_MIN',-1 )  
event.set('BACKHAUL\_RTT\_AVG',-1 )  
event.set('BACKHAUL\_RTT\_MAX',-1 )"  
}  
}  
if [BACKHAUL\_RTT\_MIN] \< 0 or [BACKHAUL\_RTT\_AVG] \< 0 or [BACKHAUL\_RTT\_MAX] \< 0{  
ruby {  
code =\> "event.set('BACKHAUL\_RTT\_MIN',-1 )  
event.set('BACKHAUL\_RTT\_AVG',-1 )  
event.set('BACKHAUL\_RTT\_MAX',-1 )"  
}  
}  
mutate {  
remove\_field =\> ["BACKHAUL\_RTT"]  
}  
}  
filter {  
if !([START\_TIME]) {  
ruby {  
code =\> "event.set('START\_TIME','0')  
event.set('START\_TIME\_MILL',0)"  
}  
}else{  
grok {  
match =\> {  
"START\_TIME" =\> [  
"%{YEAR:YEAR:int}%{MONTHNUM2:MONTHNUM2:int}%{MONTHDAY:MONTHDAY:int}%{MINUTE:HOUR1:int}%{MINUTE:MINUTE1:int}%{SECOND:SECOND:int}"  
]  
}  
}  
if !([HOUR1]) or !([MINUTE1]) or !([SECOND]) {  
ruby {  
code =\> "event.set('START\_TIME\_MILL', 0 )  
event.set('START\_TIME', '0' )"  
}  
}else{  
ruby {  
code =\> "event.set('START\_TIME\_MILL',event.get('HOUR1')\*3600000+event.get('MINUTE1')\*60000+event.get('SECOND')\*1000)"  
}  
}  
mutate {  
remove\_field =\> ["YEAR","MONTHNUM2","HOUR1","MINUTE1","SECOND"]  
}  
}  
}  
output {  
if "\_rubyexception" in [tags] {  
file {  
path =\> "var/log/logstash/errors.log"  
}  
}  
elasticsearch  
{  
hosts =\> ["localhost:9200"]  
}  
}

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 14, 2018, 11:37am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/9 "2018-03-14T11:37:18Z")

</div>

i could not upload the whole file, its about 22000 characters with 29 filters  
here you can see only 6 filters

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 14, 2018, 11:47am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/10 "2018-03-14T11:47:14Z")

</div>

That is a lot of ruby scripts, some of which look unnecessary. I am not sure how performance of the Ruby filter compares to other filters, but it might be worth replacing some of the ruby filters and see if that makes a difference.

```auto
ruby {
  code => "event.set('START_TIME','0')
    event.set('START_TIME_MILL',0)"
}

```

should as far as I can see be equivalent to:

```auto
mutate {
  add_field => {
      "START_TIME" => 0
      "START_TIME_MILL" => 0
    }
}

```

Given the size of the config there may be other things that you may be able to clean up as well in order to improve performance.

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [March 14, 2018, 11:55am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/11 "2018-03-14T11:55:49Z")

</div>

What's taking the most time in your pipeline? [Hot threads](https://www.elastic.co/guide/en/logstash/current/hot-threads-api.html) or [pipeline stats](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats) API should shed some light on that.

As Christian said, there's a lot of repetition in mutate/ruby filters that could probably be nested better.  
Also, try using [Dissect](https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-dissect.html) instead of the CSV filter, you should see some performance improvement

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 15, 2018, 9:28am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/12 "2018-03-15T09:28:49Z")

</div>

thank you paz and @Christian_Dahlqvist for your reply  
i changes my code to read Dissect as you mentioned , this change reduced the time from 14 minutes to 10 minutes,

thanks

---

<div class="post-metadata">

**Author:** ![youssef](https://avatars.discourse-cdn.com/v4/letter/y/df788c/32.png) [@youssef](https://discuss.elastic.co/u/youssef)\
**Post date:** [March 15, 2018, 11:18am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/13 "2018-03-15T11:18:27Z")

</div>

after manipulating the mutate/ruby filters, i can till that there is no difference in performance

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2018, 11:19am UTC](https://discuss.elastic.co/t/logstash-improve-time-performance/123741/14 "2018-04-12T11:19:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.