# Handling / dropping files according to previously handled files **URL:** \ **Category:** Logstash\ **Created:** [February 25, 2020, 9:18am UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799 "2020-02-25T09:18:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![AxelR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axelr/32/61269_2.png) [@AxelR](https://discuss.elastic.co/u/AxelR)\ **Post date:** [February 25, 2020, 9:18am UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/1 "2020-02-25T09:18:52Z")
I'm having an issue on how to deal with a batch of files. The lines of each file contain different pieces of information: - the first line acts as a header (with a recognizable prefix) and includes (among other things) a unique ID - the other lines (prefixed differently) must be analyzed and passed to ES IFF the unique ID in the first line was never spotted before in any other file So far, I have used the following Ruby code to store and verify the ID: ``` ruby { init => ' if @@alreadymet=nil @@alreadymet = [] end @@canGo="false" ' code => ' unless @@alreadymet&.include?(event.get("product_serial")) @@alreadymet.push(event.get("product_serial")) @@canGo="true" end event.set("can_go",@@canGo) ' } ``` However, when I further use the `can_go` attribute on the same first line, it is never evaluated to true: > if [can\_go] =~ /^true/ { [...] I'm quite new to Logstash and Ruby and I guess I'm doing something wrong... Any help appreciated 🙂 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 25, 2020, 4:22pm UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/2 "2020-02-25T16:22:24Z")
There is no reason to use a class variable (@@canGo, @@alreadymet). You can use instance variables (@canGo etc.) Once @@canGo gets set to true it will never get reset to false. It's value should depend on whether @@alreadmet includes the value of product\_serial. You are making assumptions about the ordering of events that may not be true unless you set pipeline.workers to 1 and [disable](https://github.com/elastic/logstash/issues/10938) the java execution engine. Even then you appear to be assuming that the file input will read each file in turn, and I do not think that is guaranteed. The normal way to write ``` if @@alreadymet=nil @@alreadymet = [] end ``` is ``` @@alreadymet ||= [] ``` But since this is in the init block it only gets called once, so you can use ``` @@alreadymet = [] ``` ---
**Author:** ![AxelR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axelr/32/61269_2.png) [@AxelR](https://discuss.elastic.co/u/AxelR)\ **Post date:** [February 26, 2020, 12:16pm UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/3 "2020-02-26T12:16:34Z")
> [@Badger](#): > > There is no reason to use a class variable (@@canGo, @@alreadymet). You can use instance variables (@canGo etc.) OK, I wasn't sure about Logstash behavior (ie whether an instance corresponds to a processor or the processing of a given file) > [@Badger](#): > > You are making assumptions about the ordering of events that may not be true unless you set pipeline.workers to 1 and [disable](https://github.com/elastic/logstash/issues/10938) the java execution engine. I did set pipeline workers to 1, however I didn't know about the mentioned bug. So if I understand correctly, I should use `--java-execution=false` in line command? > [@Badger](#): > > Even then you appear to be assuming that the file input will read each file in turn, and I do not think that is guaranteed. I don't understand your remark: if there is only one pipeline worker, doesn't that mean that each file will be handled after the previous one has been dealt with? Anyway, thanks for your reply! I will try to change to instance variables ASAP to see if it solves my issue. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 26, 2020, 3:05pm UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/4 "2020-02-26T15:05:27Z")
> [@AxelR](#): > > I don't understand your remark: if there is only one pipeline worker, doesn't that mean that each file will be handled after the previous one has been dealt with? That remark was not about the pipeline, it was about the input. ---
**Author:** ![AxelR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axelr/32/61269_2.png) [@AxelR](https://discuss.elastic.co/u/AxelR)\ **Post date:** [February 27, 2020, 8:24am UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/5 "2020-02-27T08:24:00Z")
OK, not sure it does solve the issue you mentioned but I used the following option in the input: > file\_sort\_by =\> "path" ---
**Author:** ![AxelR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axelr/32/61269_2.png) [@AxelR](https://discuss.elastic.co/u/AxelR)\ **Post date:** [February 27, 2020, 10:25am UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/6 "2020-02-27T10:25:35Z")
OK, I figured out what my error was: when analyzing the "other" lines (ie not the first) I simply forgot to initialize correctly the `can_go` event, duh! I just had to add this before checking `can_go` : `ruby {code => 'event.set("can_go",@@canGo)'}` @Badger: by the way, I HAD to use class variables, otherwise they would not stick from file to file... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 26, 2020, 10:25am UTC](https://discuss.elastic.co/t/handling-dropping-files-according-to-previously-handled-files/220799/7 "2020-03-26T10:25:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.