# How to index percolator field via logstash and json\_encode filter?

**URL:** <https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860>\
**Category:** Logstash\
**Created:** [July 5, 2021, 3:57pm UTC](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860 "2021-07-05T15:57:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![akb](https://avatars.discourse-cdn.com/v4/letter/a/5daacb/32.png) [@akb](https://discuss.elastic.co/u/akb)\
**Post date:** [July 5, 2021, 3:57pm UTC](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860/1 "2021-07-05T15:57:31Z")

</div>

I'm trying to index documents and populate a `percolator` field via logstash:

**Index definition**

```auto
PUT test_percolate
{
  "mappings": {
    "_doc": {
      "properties": {
        "search_name": {
          "type": "text"
        },
        "query": {
          "type": "percolator"
        }
      }
    }
  }
}

```

**Index via Kibana**

```auto
PUT test_percolate/_doc/1
{
  "search_name": "co ag",
  "query_to_percolate": {
    "match": {
      "search_name": {
        "query": "co ag",
        "operator": "and"
      }
    }
  }
}

```

This works fine, inserting via Kibana. Also works fine indexing via PHP Elastic Client.  
However, I would like to get it to work via **Logstash**

My current **logstash config** is:

```auto
input {
  ...
  jdbc {
    statement => "SELECT
        search_name
        FROM some_table
        "
    }
  }
filter {
  json_encode {
    source => "search_name"
    add_field => {
      "query" => {
        "match" => {
          "search_name" => {
            "query" => "%{search_name}"
            "operator" => "and"
          }
        }
      }
    }
  }
}
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "<host_name>"
  "index" => "test_percolate"
  "document_type" => "_doc"
  }
}

```

I've tried various notations, but I'm not getting anywere with it and keep getting the following error:

```auto
[WARN] 2021-07-05 17:52:17.013 [[main]>worker1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test_percolate", :routing=>nil, :_type=>"_doc"}, {"search_name"=>"\"ag & co.\"", "@version"=>"1", "@timestamp"=>2021-07-05T15:52:16.578Z, "query"=>"[\"match\", {\"search_name\"=>{\"query\"=>\"\"ag & co.\"\", \"operator\"=>\"and\"}}]"}], :response=>{"index"=>{"_index"=>"test_percolate", "_type"=>"_doc", "_id"=>"oOJed3oB_2KydUX5w3xK", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parsing_exception", "reason"=>"[_na] query malformed, must start with start_object", "line"=>1, "col"=>180}}}}}

```

Does anyone have any pointers on how to index into `percolator` field via logstash?  
Thanks!

---

<div class="post-metadata">

**Author:** ![akb](https://avatars.discourse-cdn.com/v4/letter/a/5daacb/32.png) [@akb](https://discuss.elastic.co/u/akb)\
**Post date:** [July 6, 2021, 8:26am UTC](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860/2 "2021-07-06T08:26:44Z")

</div>

After much trial & error, I finally got it working with the following config.

```auto
input {
  ...
  jdbc {
    statement => "SELECT
        search_name
        FROM some_table
        "
    }
  }
filter {
  json_encode {
    source => "search_name"
    target => "escaped_search_name"
  }
  mutate {
    add_field => {
      "[query]" => '{
        "match" => {
          "search_name" => {
            "query" => %{escaped_search_name}
            "operator" => "and"
          }
        }
      }'
    }
  }
  json {
    source => "query"
    target => "query"
  }
  mutate {
    remove_field => ["escaped_search_name"]
  }
}
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "<host_name>"
  "index" => "test_percolate"
  "document_type" => "_doc"
  }
}

```

Turns out, I had it backwards with the `json` filter and `json_encode` filter the whole time. To pass on a structured json, the `json` filter should be used.

**The filter parts explained:**

1. `json_encode` constructs escaped / encoded input before it used in the construction of json string
2. `mutate add_field`: create a new field `[query]` with its _value being a json string_ (` '{...}'` ). It is important to have the field as` [field_name]`, otherwise the next step to parse the string into json object doesn't work
3. `json`: parse the string representation into a json object
4. `mutate remove_field`: get rid of the escaped field, if it shouldn't be indexed

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 3, 2021, 8:27am UTC](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860/3 "2021-08-03T08:27:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.