# Trouble using elasticsearch filter plugin

**URL:** <https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981>\
**Category:** Logstash\
**Created:** [September 10, 2017, 5:36pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981 "2017-09-10T17:36:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![cito.ets](https://avatars.discourse-cdn.com/v4/letter/c/4491bb/32.png) [@cito.ets](https://discuss.elastic.co/u/cito.ets)\
**Post date:** [September 10, 2017, 5:36pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/1 "2017-09-10T17:36:28Z")

</div>

Hi,

A bit new into SSL/TLS. I do have managed to get Logstash to work with Searchguard/Elasticsearch (elasticsearch output) on a test server but when trying to use the elasticsearch filter plugin I get a ConnectionFailed warning in Logstash but nothing from Elasticsearch.

logstash.conf

```
filter {
...
    elasticsearch {
      hosts => ["https://127.0.0.1:9200"]
      index => "logstash-index-ref"
      user => "logstash"
      password => " *****"
      ssl => true
      query => "BusinessEmail:%{user}"
      fields => { "FirstName" => "FirstName" }
    }
...
}
...
output {
  elasticsearch {   
    hosts => ["https://127.0.0.1:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "logstash"
    password => " *****"
    ssl => true
    ssl_certificate_verification => false
    truststore => "/etc/elasticsearch/truststore.jks" 
    truststore_password => "changeit"
  }
  stdout { codec => rubydebug }
}

```

sg\_roles.yml  
sg\_logstash:  
cluster:  
- indices:admin/template/get  
- indices:admin/template/put  
- indices:data/write/bulk\*  
indices:  
'logstash-_':  
'_':  
- CRUD  
- CREATE\_INDEX  
'_beat_':  
'\*':  
- CRUD  
- CREATE\_INDEX

I'm using the search-guard demo. Security works across Logstash-\>ES-\>Kibana, except I really couldn't get the elasticsearch filter plugin to work.

logstash-plain.log  
[2017-09-10T13:31:19,688][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=\>"logstash-index-ref", :query=\>"BusinessEmail:-", :event=\>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:47.805788Z **_.\*\*\*.\*\*\*._** -  
2017-09-07T04:37:47.808586Z 3.798ms  
HTTP/1.1 401 Unauthorized  
Content-Type: application/json;charset=UTF-8

```
{
  "responseCode": 401
}
2017-09-07T04:37:47.809702Z 4.914ms

, :error=>#<Faraday::ConnectionFailed>}
[2017-09-10T13:31:19,736][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-index-ref", :query=>"BusinessEmail:test@test.com", :event=>2017-09-10T12:30:54.824Z ubuntu 2017-09-07T04:37:48.216151Z ***.***. ***.*** test@test.com
2017-09-07T04:37:48.852590Z 637.439ms
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8

{
  "responseCode": 200
}
2017-09-07T04:37:48.853740Z 638.589ms

, :error=>#<Faraday::ConnectionFailed>}

```

Anyone have an idea? The elasticsearch-filter-plugin doc doesn't have the other security options in their elasticsearch output API.

Cinto

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 10, 2017, 5:42pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/2 "2017-09-10T17:42:00Z")

</div>

Don't you need to put security certificate path in the filter? Did you try ca\_file setting?

---

<div class="post-metadata">

**Author:** ![cito.ets](https://avatars.discourse-cdn.com/v4/letter/c/4491bb/32.png) [@cito.ets](https://discuss.elastic.co/u/cito.ets)\
**Post date:** [September 11, 2017, 4:08am UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/3 "2017-09-11T04:08:53Z")

</div>

Thanks. I have tried trial-and-erroring with the following ca\_file options (not sure which one to put in, so I just tried each one):

Using self-signed certificates (listed in keystore.jks)  
ca\_file =\> "/dir/server.cer"  
ca\_file =\> "/dir/server.pem"

And the jks files from the demo  
ca\_file =\> "/etc/elasticsearch/trustore.jks"  
ca\_file =\> "/etc/elasticsearch/keystore.jks"  
ca\_file =\> "/etc/elasticsearch/kirk.jks"

Nothing worked and it just produces similar errors as previously pasted.

Adding in some details:  
Elastic Stack - 5.5.0  
Ubuntu 16.0.4  
ES and Logstash are in the same machine  
-I'm also able to curl --insecure 127.0.0.1 without setting a certificate file

---

<div class="post-metadata">

**Author:** ![cito.ets](https://avatars.discourse-cdn.com/v4/letter/c/4491bb/32.png) [@cito.ets](https://discuss.elastic.co/u/cito.ets)\
**Post date:** [September 11, 2017, 8:35am UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/4 "2017-09-11T08:35:08Z")

</div>

Still stuck, still hopeful. Could this issue be related in any way to:

> **Fixes in master and 6.0**  
> Elasticsearch Filter: Support ca\_file setting when using https URI in hosts parameter (#58).

> **[Logstash Lines: Introducing a benchmarking tool for Logstash](https://www.elastic.co/blog/logstash-lines-2017-08-01)**

> <https://github.com/logstash-plugins/logstash-filter-elasticsearch/issues/58>
>
> I'm trying to use the elastic cloud as my filter source. I have a config like:
> …
> 
> filter {
> elasticsearch {
> hosts =\> \["https://FOUND\_SERVER.us-east-1.aws.found.io:9243"\]
> ssl =\> true
> user =\> "elastic"
> password =\> "changeme"
> }
> }
> 
> This doesn't work. I tried also uninstalling the elasticsearch input plugin, which was pinning elasticsearch ruby client library to 1.0 so that installing the elasticsearch filter would install the 5.0 version of the ruby client library, but it didn't change anything.
> 
> Using http and ssl =\> false and port 9200 works fine for accessing the cluster so it does appear to be some sort of bug in the filter code.

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 11, 2017, 10:03am UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/5 "2017-09-11T10:03:18Z")

</div>

You seem to have figured out! The links seem relevant.

However, I would request someone senior in this matter to comment on this.

---

<div class="post-metadata">

**Author:** ![cito.ets](https://avatars.discourse-cdn.com/v4/letter/c/4491bb/32.png) [@cito.ets](https://discuss.elastic.co/u/cito.ets)\
**Post date:** [September 11, 2017, 10:38am UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/6 "2017-09-11T10:38:02Z")

</div>

Is there a viable workaround for enrichment of data? (Without JDBC) Like, would creating 2-3 translate filters instead as lookup be safe in terms of performance?

Or should I just wait out on ES 6?

---

<div class="post-metadata">

**Author:** ![karanshah](https://avatars.discourse-cdn.com/v4/letter/k/3be4f8/32.png) [@karanshah](https://discuss.elastic.co/u/karanshah)\
**Post date:** [September 27, 2017, 12:30pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/7 "2017-09-27T12:30:45Z")

</div>

Hi @cito.ets , I am facing the same issue with elasticsearch filter. I compiled a gem file from master of [logstash-filter-elasticsearch](https://github.com/logstash-plugins/logstash-filter-elasticsearch) which contains the bug fix.  
I used a config with SSL false and ca\_file pointing to PEM file of my CA but the configuration still fails with below error  
`[2017-09-27T13:16:01,437][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-XXX-task-his-*", :query=>{"query"=> {"bool"=>{"must_not"=>{"exists"=>{"field"=>"srvr_status"}}, "must"=>[{"match"=> {"srvr_user_name.keyword"=>"XXX"}}]}}, "_source"=>["row_id", "srvr_start_ts", "srvr_end_ts"]}, :event=>2017-09-27T12:16:00.378Z %{host} %{message}, :error=># <Faraday::SSLError>`

On ElasticSearch server I get below error  
`[2017-09-27T12:39:01,315][WARN][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ES-DEV-NODE-1] caught exception while handling client http traffic, closing connection [id: 0x6a69b4ce, L:0.0.0.0/0.0.0.0:9203 ! R:/10.33.15.194:38707] io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`

Will keep you posted if I make progress on getting it to work.

---

<div class="post-metadata">

**Author:** ![karanshah](https://avatars.discourse-cdn.com/v4/letter/k/3be4f8/32.png) [@karanshah](https://discuss.elastic.co/u/karanshah)\
**Post date:** [September 28, 2017, 2:47pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/8 "2017-09-28T14:47:24Z")

</div>

Hi @cito.ets, my configuration is finally working based on the new GEM file that I used. The SSL error was resolved by setting ca\_file to path of .cer file that contained chained cert for my intermediate and root CA. Previously I was using only Intermediate CA.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2017, 2:47pm UTC](https://discuss.elastic.co/t/trouble-using-elasticsearch-filter-plugin/99981/9 "2017-10-26T14:47:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.