# Slow Pipeline after adding output parameter

**URL:** <https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850>\
**Category:** Logstash\
**Created:** [July 19, 2016, 9:49am UTC](https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850 "2016-07-19T09:49:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![NicoYUE](https://avatars.discourse-cdn.com/v4/letter/n/96bed5/32.png) [@NicoYUE](https://discuss.elastic.co/u/NicoYUE)\
**Post date:** [July 19, 2016, 9:49am UTC](https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850/1 "2016-07-19T09:49:57Z")

</div>

Hello,

I'm using Logstash to retrieve data from a SQL database to push into Elasticsearch.

I had some good result with around 1000 inputs per second into my cluster.

Next step was to put an id on my documents so that, when my pipeline restarts with the schedule parameter, it wouldn't send duplicates to my Cluster.

Since I added this document\_id in the output, my pipeline is only sending 2 entries per second ... (I emptied my cluster before running again) it even stops at some points, around 250 docs.

Here's my conf file, all worked before I add the simple line document\_id in output.

input {  
jdbc {  
jdbc\_connection\_string =\> "jdbc:jtds:sqlserver://ORCHESTRATOR1:60113;DatabaseName=ePO\_ORCHESTRATOR1;domain=ORCHESTRATOR1;socketKeepAlive=true"  
jdbc\_user =\> "Administrator"  
jdbc\_password =\> "xxx"  
jdbc\_validate\_connection =\> true  
jdbc\_driver\_library =\> "/root/jtds-1.3.1.jar"  
jdbc\_driver\_class =\> "Java::net.sourceforge.jtds.jdbc.Driver"  
statement =\> "SELECT \* FROM dbo.EPOEvents WHERE DATEPART(yy, receivedutc) \>= '2015'"  
}  
}

filter{

mutate{  
convert =\> ["receivedutc", "string"]  
}

date {  
match =\> ["receivedutc", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]  
}  
ruby{

code =\>  
"event['o1'] = (event['sourceipv4']+2147483648)/16777216;  
event['ip'] = (event['sourceipv4']+2147483648)%16777216;  
event['o2'] = event['ip']/65536;  
event['ip2'] = event['ip']%65536;  
event['o3'] = event['ip2']/216;  
event['o4'] = event['ip2']%216;  
event['src\_ipv4'] = (event['o1']).to\_s+'.'+(event['o2']).to\_s+'.'+(event['o3']).to\_s+'.'+(event['o4']).to\_s;  
event['o1'] = (event['analyzeripv4']+2147483648)/16777216;  
event['ip'] = (event['analyzeripv4']+2147483648)%16777216;  
event['o2'] = event['ip']/65536;  
event['ip2'] = event['ip']%65536;  
event['o3'] = event['ip2']/216;  
event['o4'] = event['ip2']%216;  
event['analyzer\_ipv4'] = (event['o1']).to\_s+'.'+(event['o2']).to\_s+'.'+(event['o3']).to\_s+'.'+(event['o4']).to\_s;  
event['o1'] = (event['targetipv4']+2147483648)/16777216;  
event['ip'] = (event['targetipv4']+2147483648)%16777216;  
event['o2'] = event['ip']/65536;  
event['ip2'] = event['ip']%65536;  
event['o3'] = event['ip2']/216;  
event['o4'] = event['ip2']%216;  
event['target\_ipv4'] = (event['o1']).to\_s+'.'+(event['o2']).to\_s+'.'+(event['o3']).to\_s+'.'+(event['o4']).to\_s"

remove\_field =\> ["o1", "o2", "o3", "o4", "ip", "ip2"]

}  
mutate {  
remove\_field =\> ["autoguid", "agentguid", "sourcehostname", "thetimestamp", "analyzeripv4", "sourceipv4", "targetipv4","analyzeripv6", "sourceipv6", "targetipv6"]  
}  
}

output {

elasticsearch {  
hosts =\> ["xxx:9200"]  
index =\> "orchestrator-event-%{+YYYY.MM.dd}"  
document\_id =\> "autoid"

}  
}

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![NicoYUE](https://avatars.discourse-cdn.com/v4/letter/n/96bed5/32.png) [@NicoYUE](https://discuss.elastic.co/u/NicoYUE)\
**Post date:** [July 19, 2016, 9:59am UTC](https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850/2 "2016-07-19T09:59:48Z")

</div>

Just noticed there are only one document per index which is weird, autoid I set on document\_id is unique to every events in the database.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 21, 2016, 7:58am UTC](https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850/3 "2016-07-21T07:58:45Z")

</div>

Is the 'autoid' field populated in you database query? In order to get the value of this field used as document id, you need to change the document\_id specification to: `document_id => "%{autoid}"`. With the current configuration all records get the string 'autoid' as key, which results in a very large number of updates of the same document.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:47am UTC](https://discuss.elastic.co/t/slow-pipeline-after-adding-output-parameter/55850/4 "2017-07-06T04:47:05Z")

</div>

Closed on July 6, 2017, 4:47am UTC