# Refresh in pipeline: output and/or filter **URL:** \ **Category:** Logstash\ **Created:** [July 7, 2021, 8:26am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044 "2021-07-07T08:26:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![itokai](https://avatars.discourse-cdn.com/v4/letter/i/db5fbb/32.png) [@itokai](https://discuss.elastic.co/u/itokai)\ **Post date:** [July 7, 2021, 8:26am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/1 "2021-07-07T08:26:58Z")
Hi, in logstash pipeline consecutive documents are consumed from RabbitMQ. Sometimes need to perform lookup on already imported documents via _filter-\>elasticsearch-\>query\_template_. However, lookup doesn't query the latest state, as refresh is required! Have tried to explicitly perform refresh via _http_ plugin on each node of the cluster, before lookup ```auto http { verb => "POST" url => "https://elasticX:9200/index_for_refresh/_refresh" user => "user" password => "password " cacert => "/etc/logstash/certs/elastic-ca.pem" } ``` but no it is not consistent and can't rely on it. [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html) How to perform steady refresh either in **filter** or in **output** , so latest cluster state serves lookup? Regards ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [July 15, 2021, 12:25am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/2 "2021-07-15T00:25:23Z")
That feels a little like using a hammer on a screw. Maybe we can find another approach? Can you share a little what those documents are, why you need the lookup and how that query looks like, and what the end goal is? ---
**Author:** ![itokai](https://avatars.discourse-cdn.com/v4/letter/i/db5fbb/32.png) [@itokai](https://discuss.elastic.co/u/itokai)\ **Post date:** [July 16, 2021, 8:54am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/3 "2021-07-16T08:54:20Z")
Thanks xeraa, split into 2 separate (independent) processes: 1. continuously importing into index, where RabbitMQ is configured as input 2. periodically performing lookup, where elaticsearch index (above) is configured as input. If lookup returned results, than saving the enriched document in new index; otherwise it's examined in next iteration That does it for now, let me know if you have an advice or better insight please Regards ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [July 20, 2021, 12:13am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/4 "2021-07-20T00:13:26Z")
It sounds relatively similar to the [enrich ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html) — maybe that's an alternative? It runs within Elasticsearch and won't need Logstash or a queue; though it has some limitations around updating the lookup index. It might still be a better fit? Also I'm not sure I follow this part on the `_refresh` API: > [@itokai](#): > > is not consistent and can't rely on it. While not recommended for production because of the performance overhead, this should do the right thing. ---
**Author:** ![itokai](https://avatars.discourse-cdn.com/v4/letter/i/db5fbb/32.png) [@itokai](https://discuss.elastic.co/u/itokai)\ **Post date:** [July 20, 2021, 1:45pm UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/5 "2021-07-20T13:45:27Z")
Thanks xeraa, have tried _Enrich policy_, but _elasticsearch filter_ fits better for this use-case, because retrieved results can be conditionally handled on custom way. _\_refresh API_ doesn't synchronously execute (wait to be refreshed) within same logstash pipeline, so latest could not be sequentially collected in next lines of pipeline code. As you wrote, it's not recommended to attempt to use ElasticSearch in transactional manners. Regards ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [July 20, 2021, 2:17pm UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/6 "2021-07-20T14:17:42Z")
Interesting. `_refresh` is generally a blocking call. What's your setting for `pipeline.workers` and does 1 make a difference? But all of this feels very much like a workaround. Also calling the `_refresh` for every single document in a batch. ---
**Author:** ![itokai](https://avatars.discourse-cdn.com/v4/letter/i/db5fbb/32.png) [@itokai](https://discuss.elastic.co/u/itokai)\ **Post date:** [August 4, 2021, 6:54am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/7 "2021-08-04T06:54:33Z")
Thanks xeraa, _\_refresh_ had to go away due to bigData performance (as you confirmed), so going with 2 separate processes described above Regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 1, 2021, 6:55am UTC](https://discuss.elastic.co/t/refresh-in-pipeline-output-and-or-filter/278044/8 "2021-09-01T06:55:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.