# Using percolate in a Logstash filter?

**URL:** <https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381>\
**Category:** Logstash\
**Created:** [February 24, 2017, 10:07am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381 "2017-02-24T10:07:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![es435](https://avatars.discourse-cdn.com/v4/letter/e/bc79bd/32.png) [@es435](https://discuss.elastic.co/u/es435)\
**Post date:** [February 24, 2017, 10:07am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/1 "2017-02-24T10:07:30Z")

</div>

I have an index of percolator queries on Elasticsearch and I'm trying to write a Logstash config file with a filter that will allow me to percolate documents against those queries as I index them, sending an alert if there's a match.  
Researching this I've come across suggestions that it's possible to do using the elasticsearch plugin but I'm not sure what the syntax would be to reference both the current document being indexed and the percolate index - if that makes sense?  
I'm using ES 5.2 - so percolate as a query type rather than the standalone API. Any advice?  
Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 24, 2017, 10:18am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/2 "2017-02-24T10:18:59Z")

</div>

It should "just work" as it's another query type as you mention.

That said, I have not tried 😛

---

<div class="post-metadata">

**Author:** ![es435](https://avatars.discourse-cdn.com/v4/letter/e/bc79bd/32.png) [@es435](https://discuss.elastic.co/u/es435)\
**Post date:** [February 24, 2017, 11:03am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/3 "2017-02-24T11:03:02Z")

</div>

Hi Mark,  
Thanks for your reply - glad it sounds feasible as a concept.  
I've had a go at the config file and I'm getting an error message - 'logstash.agent fetched an invalid config' at the Elasticsearch bit. The code I'm trying is:

```
elasticsearch {
    query => "{ percolate { hosts => ["http://localhost:9200"], index => "data-search", type => "queries", id => "%{id}" }"
}

```

I'm very new to Logstash and Elasticsearch so no idea if I'm thinking along the right lines..

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 25, 2017, 12:43am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/4 "2017-02-25T00:43:37Z")

</div>

That syntax doesn't look right, check the docs for a guide - [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)

---

<div class="post-metadata">

**Author:** ![es435](https://avatars.discourse-cdn.com/v4/letter/e/bc79bd/32.png) [@es435](https://discuss.elastic.co/u/es435)\
**Post date:** [February 27, 2017, 10:36am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/5 "2017-02-27T10:36:18Z")

</div>

Using the docs I think I've worked out the syntax I need - the problem I'm having is that the document I'm passing through to the percolator hasn't been indexed yet. Given that I'm structuring my query as follows:

```
elasticsearch {
    hosts => ["http://localhost:9200/data-search/queries"]
    user => "elastic"
    password => ""
    index => "data-search"
    query => '"percolate" : { "field" : "query", "document_type" : "doctype", "document" : { "" }}'
}

```

Can anybody point me in the right direction for how to refer to the document currently being indexed in the "document" field?  
Thanks!

---

<div class="post-metadata">

**Author:** ![pemontto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pemontto/32/3908_2.png) [@pemontto](https://discuss.elastic.co/u/pemontto)\
**Post date:** [March 16, 2017, 12:25am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/6 "2017-03-16T00:25:36Z")

</div>

I tried using the full query DSL like you have and substituted fields into the query using the sprintf format so it looked something like this, but don't know how you could get the whole event into the query without manually defining the fields:

```
elasticsearch {
  hosts => ["test-es:9204"]
  index => "news_percolator"
  fields => { "_id" => "matches" }
  query => '"percolate" : { "field" : "query", "document_type" : "doctype", "document" : { "title": "%{title}", "summary": "%{summary}" }}'
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2017, 12:25am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381/7 "2017-04-13T00:25:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.