Data ingestion error

Elastic Stack Logstash
1

When I perform the ingestion everything seems to be fine, the information is displayed correctly in kibana but during the process the following errors appear

[ERROR] 2024-08-23 15:08:25.370 [[main]>worker2] ruby - Ruby exception occurred: undefined method `each' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):4:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:158:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:176:in `block in multi_filter'", "org/jruby/RubyArray.java:1981:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:173:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in `block in start_workers'"]}
[ERROR] 2024-08-23 15:08:25.373 [[main]>worker2] ruby - Ruby exception occurred: undefined method `each' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):4:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:158:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:176:in `block in multi_filter'", "org/jruby/RubyArray.java:1981:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:173:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in `block in start_workers'"]}
[ERROR] 2024-08-23 15:08:25.376 [[main]>worker2] ruby - Ruby exception occurred: undefined method `each' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):4:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:158:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:176:in `block in multi_filter'", "org/jruby/RubyArray.java:1981:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:173:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in `block in start_workers'"]}

It seems that everything is fine but I don't understand these errors

The .conf file that I use for ingestion into logstash is the following

input {
  file {
    path => "/home/lab/BFSALIDA/*.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
	pattern => "^<\?xml"
        negate => "true"
        what => "previous"
	auto_flush_interval => 5
    }
  }
}

filter {
  grok { match => { "message" => "<body>%{GREEDYDATA:theHTML}</body>" } }

  mutate { gsub => [ "message", '<html xmlns="[^"]*"', '<html' ] }
  xml { 
    source => "message" 
    store_xml => false 
    remove_field => [ "message" ] 
    xpath => {
      '/html/head/meta[@name="dcterms:created"]/@content' => "Create_Date"
      '/html/head/meta[@name="Message:From-Email"]/@content' => "Message_from"
      '/html/head/meta[@name="Message-To"]/@content' => "Message_To"
      '/html/head/meta[@name="dc:title"]/@content' => "Titulo_email"
      '/html/head/meta[@name="resourceName"]/@content' => "eml_origen"
    }
  
  
  }
    ruby {
        code => '
            meta = event.remove("[parsed][head][meta]")
            meta.each { |x|
                key = x["name"]
                if key =~ /^Message:Raw-Header:/
                    newKey = key.sub(/^Message:Raw-Header:/, "")
                    newKey = "[headers][#{newKey}]"
                else
                    newKey = key.sub(/^.*:/, "")
                    newKey = "[otherStuff][#{newKey}]"
                end
                event.set(newKey, x["content"])
            }
        '
    }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "emails"
  }
}
2

Ruby exception occurred: undefined method each' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):4:in block in register'",

The error message says that you are calling .each for a nil value. So meta must be nil -- there are events for which [parsed][head][meta] does not exist.

In the error message "(ruby filter code):4:in tells you it is somewhere close to the fourth line in the code => block (although it never seems to get that number quite right).

ETA: You could avoid the exception using meta&.each { |x|, or the more verbose meta && meta.each { |x|, or even a begin/rescue block.

1 Like
3

Just entering an & between meta and .each. Is it enough? All the xml files are uploaded. I see that this only happens with some xml files, not with all of them. I imagine that some will have empty fields and others will not. Am I right?

4

In this case, yes. Both meta&.each and meta && meta.each are ways of saying "If meta is not nil then call meta.each". If meta.each is not called then nothing will get populated within [headers] or [otherStuff].