# Bad File descriptor error with Large dictionary file (\>10MB) in Logstash

**URL:** <https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225>\
**Category:** Logstash\
**Created:** [July 24, 2016, 5:46am UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225 "2016-07-24T05:46:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nishant\_goel](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@nishant\_goel](https://discuss.elastic.co/u/nishant_goel)\
**Post date:** [July 24, 2016, 5:46am UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225/1 "2016-07-24T05:46:06Z")

</div>

I have multiple log messages in a file which I am processing using logstash filter plugins. Then, the filtered logs are getting sent to elasticsearch.

There is one field called addID in a log message. I want to drop all the log messages which have a particular addID present. These particular addIDS are present in a ID.txt file.

If the addID of a log message matches with any of the addIDs present in the ID.txt file, that log message should be dropped. I am using using a ruby filter for achieving this.

**Scenario: Issue is that if the dictionary file that I a using is in MBs then, logstash hangs and I get Bad file descriptor error when I manually stops the pipeline. However, if I use file in KBs, everything works fine.**

I have tried changing the LS\_HEAP\_SIZE to 4g. Nothing worked for me.

Could anyone help me in achieving this?

@magnusbaeck @warkolm  
PLease help me Sir

Below is my config file.

```
input {

    file {
        path => "/Users/jshaw/logs/access_logs.logs
        ignore_older => 0
    }
}

filter {

    grok {

        patterns_dir => ["/Users/jshaw/patterns"]
        match => ["message", "%{TIMESTAMP:Timestamp}+{IP:ClientIP}+{URI:Uri}"]

    }

    kv{
        field_split => "&?"
        include_keys => ["addID"]
        allow_duplicate_values => "false"
        add_field => { "IS_BAD_IP" => "false" } 
    }

    if [ClientIP] {
         ruby{
             code => 'if File.open("/Users/jsaw/mapping/badIP.txt").lines.any?{|line|line.include?(event["ClientIP"])}
                  event["IS_BAD_IP"] = "true"
             end'

         }   

         if "true" in [IS_BAD_IP]{
              drop { }
        }     

 }

output {

     elasticsearch{
         hosts => ["localhost:9200"]
     }
}
```

---

<div class="post-metadata">

**Author:** ![jpcarey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpcarey/32/46668_2.png) [@jpcarey](https://discuss.elastic.co/u/jpcarey)\
**Post date:** [July 24, 2016, 11:28am UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225/2 "2016-07-24T11:28:52Z")

</div>

[Drop filter not working](https://discuss.elastic.co/t/drop-filter-not-working/55940/4?u=jpcarey) ?

---

<div class="post-metadata">

**Author:** ![nishant\_goel](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@nishant\_goel](https://discuss.elastic.co/u/nishant_goel)\
**Post date:** [July 24, 2016, 6:31pm UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225/3 "2016-07-24T18:31:12Z")

</div>

@jpcarey

I have tried the translate filter also but I could not achieve my goal using a large dictionary. Do you know any workaround.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225/4 "2017-07-06T04:46:38Z")

</div>