# Logstash pipeline using elasticsearch filter experiencing low performance
**URL:** https://discuss.elastic.co/t/logstash-pipeline-using-elasticsearch-filter-experiencing-low-performance/172739
**Category:** Logstash
**Created:** [March 18, 2019, 9:55am UTC](https://discuss.elastic.co/t/logstash-pipeline-using-elasticsearch-filter-experiencing-low-performance/172739 "2019-03-18T09:55:55Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@JeroenK](https://discuss.elastic.co/u/JeroenK)
#### Post date: [March 18, 2019, 9:55am UTC](https://discuss.elastic.co/t/logstash-pipeline-using-elasticsearch-filter-experiencing-low-performance/172739/1 "2019-03-18T09:55:55Z")
Hi there,
I'm probably having some infrastructural problem, but I sure can use some help in finding out what I need to look at and fix.
I'm trying to 'enrich' data from one index with data from another index using logstash. My indexing speed is smaller than 1 d/s, which I think could be better 😉
I'm using logstash in a docker on a local macbook (which is probably a reason for low performance, but I reckon that even in this setup, 1 d/s is way to slow and can be better).
I'm using a cloud trial:
- 2 \* AWS.data.highio.i3, 4GB
- 1 \* AWS.master.R4 1GB tiebreaker
Indexing from a csv from local machine using the same logstash docker performs at around 200 documents per second. I'm fine with that
This is my pipeline, it actually works, but as mentioned: really sloooow:
```
input {
elasticsearch {
hosts => ["https://xxxxxxx"]
user => "xxxx"
password => "xxxx"
docinfo => true
query => '{
"query": {
"bool": {
"must": [
{ "exists": { "field": "xxxxx" }}
],
"filter": [
{ "range": {
"start_dtime": {
"gte": "2019-03-12",
"lte": "2019-03-13"
}
}
}
]
}
}
}'
}
}
filter {
mutate {
remove_field => ["@version", "host", "message", "path"]
add_field => {
"xxxx_date" => "unknown"
"xxxxx_status" => "unknown"
"xxxx_name" => "unknown"
}
}
elasticsearch {
hosts => ["xxxxxxxx"]
user => "xxxx"
password => "xxx"
index => "xxxxxxxx*"
query => "xxxxx: %{[xxxxx]}"
fields => {
"xxxx_date" => "xxxx_date"
"xxxx_status" => "xxxx_status"
"xxxx_name" => "xxxx_name"
}
}
}
output {
elasticsearch {
hosts => ["xxxxxxxx"]
index => "%{[@metadata][_index]}"
action => "update"
document_id => "%{[@metadata][_id]}"
user => "xxxx"
password => "xxxx"
}
}
```
My logstash configuration:
```
pipeline:
workers: 5
batch:
size: 1000
delay: 50
```
Index settings
```
"settings" : {
"index" : {
"creation_date" : "1552728048556",
"number_of_shards" : "1",
"number_of_replicas" : "1",
```
Maybe anyone can give me a hint where to look or tweak this so it will be running a bit faster?
Thanks!
Jeroen
---
### Author:  [@JeroenK](https://discuss.elastic.co/u/JeroenK)
#### Post date: [April 14, 2019, 2:58pm UTC](https://discuss.elastic.co/t/logstash-pipeline-using-elasticsearch-filter-experiencing-low-performance/172739/2 "2019-04-14T14:58:59Z")
Anyone any tips?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 12, 2019, 2:59pm UTC](https://discuss.elastic.co/t/logstash-pipeline-using-elasticsearch-filter-experiencing-low-performance/172739/3 "2019-05-12T14:59:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.