# Filter Plugin: Elasticsearch

**URL:** <https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144>\
**Category:** Logstash\
**Created:** [March 11, 2016, 12:21pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144 "2016-03-11T12:21:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Christian16](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@Christian16](https://discuss.elastic.co/u/Christian16)\
**Post date:** [March 11, 2016, 12:21pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/1 "2016-03-11T12:21:05Z")

</div>

Hello,  
My main goal is to get a tag (TRANSFER ID) from a previous log event, and i would like to include this tag in my current log event. So I decided to use the "Elasticsearch Filter" plugin but I'm having trouble with it:

```auto
Failed to query elasticsearch for previous event { <
........
.......
SOME LOGS EVENTS
.......
.......
 >}>>, :error=>#<Elasticsearch::Transport::Transport::Errors::MovedPermanently: [301] <HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
>, :level=>:warn} 

```

Here my code:

```auto
input {
    file{
        path => ["/home/christian/Documents/ELK/logs-interop/oma-interop-app.log-20160207"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
  stdin{}
}
filter {
    grok {
        patterns_dir => "./patterns"
        match => ["message", "%{DEFAULT}"]
    }
   elasticsearch {
        hosts => ["127.0.0.1"]
        query => "session_id:2" 
        fields => ["TRANSFER ID", "test_id"]
   }
}
output {
    stdout {
        codec => rubydebug
    } 
    elasticsearch {
        hosts => "127.0.0.1"
        index => "logstash-interop-app"
    }
} 

```

May you help me please.

Sincerely,  
Chris

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 12, 2016, 1:50am UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/2 "2016-03-12T01:50:17Z")

</div>

Looks like something is proxying Elasticsearch which the plugin cannot handle.

---

<div class="post-metadata">

**Author:** ![Christian16](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@Christian16](https://discuss.elastic.co/u/Christian16)\
**Post date:** [March 12, 2016, 7:51pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/3 "2016-03-12T19:51:44Z")

</div>

Yes, with proxy disabled it works. Is there a way to configure the plugin to handle the proxy ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 12, 2016, 8:52pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/4 "2016-03-12T20:52:49Z")

</div>

Not according to the docs.

---

<div class="post-metadata">

**Author:** ![Christian16](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@Christian16](https://discuss.elastic.co/u/Christian16)\
**Post date:** [March 14, 2016, 10:20am UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/5 "2016-03-14T10:20:01Z")

</div>

Thank you warkolm for your answers.

Else, I got another issue with this plugin.

To example I got this log in a file:

```
2016-02-06T04:45:48.186Z INFO iop_mg_1454733936.741_433 task Interop\TransferService >> OUTGOING REQUEST to telma MMITransaction {"transaction_Id":"iop_mg_1454733915.236_784"}
2016-02-06T04:46:01.092Z INFO iop_mg_1454733936.741_433 task Sending SMS to=000000000, encoding=UTF-8, text=ok. 

```

I want the second event "Sending SMS" retrieve the "transaction\_Id" tag from the previous event thanks to session\_id(iop\_mg\_1454733936.741\_433).

Here my filter :

```
  elasticsearch {
                hosts => ["127.0.0.1"]
                query => "session_id:%{session_id} AND _exists_:transaction_Id"
                fields => ["transaction_Id", "verif_id"]
           }

```

When i run logstash, the transaction\_id is not found. I think is because logstash take to much time to add the first event in elasticsearch and when it is trying to filter the second log event, it can't find nothing because the first log event is not yet saved in elasticsearch.

I tried not to use a file as input but stdin{} to insert my event log 1 by 1 using an interval of 2 sec and it works. But if I use a short interval to example 1sec it doesn't work.

How can I solve this ?

Sorry for my English, hope you have understood my issue.

Sincerely,  
Chris.

---

<div class="post-metadata">

**Author:** ![Christian16](https://avatars.discourse-cdn.com/v4/letter/c/a9a28c/32.png) [@Christian16](https://discuss.elastic.co/u/Christian16)\
**Post date:** [March 14, 2016, 2:27pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/6 "2016-03-14T14:27:30Z")

</div>

i found a solution to retrieve data from a previous event using Ruby Fitler.

```
   if[transaction_Id] {
        ruby {  
            init => "@@map = {}"
            code => "
                @@map[event['verif_id']] = event['transaction_Id']
                @@map[event['session_id']] = event['session_id']
            "
        }
   } else if[classname] == "SendingSMS" {
        ruby{
            code => "
                if event['session_id'] == @@map[event['session_id']] 
                then
                    event['transaction_Id'] = @@map[event['verif_id']] 
                end"
        }
   }

```

Sincerely,  
Chris.

---

<div class="post-metadata">

**Author:** ![Leandro\_Sampaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandro_sampaio/32/18409_2.png) [@Leandro\_Sampaio](https://discuss.elastic.co/u/Leandro_Sampaio)\
**Post date:** [May 21, 2017, 3:39pm UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/7 "2017-05-21T15:39:51Z")

</div>

Did you make it work??

I made the samething and I take this error:

[logstash.agent] Error in reactor loop escaped: Bad file descriptor - Bad file descriptor (Errno::EBADF)

Could you help me??

And sorry my english... I 'm learning

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:26am UTC](https://discuss.elastic.co/t/filter-plugin-elasticsearch/44144/8 "2017-07-06T04:26:27Z")

</div>