Logstash Ruby error when trying to rename a field the has a hyphen "-" in the value

[Bryan_Hamilton]

I have events with nested fields and am trying and am trying the reduce the depth of some nested fields to match ecs and discarding the fields I don't need. Renaning of all other fields word except for one which causes a logstash ruby error.

The event hierarchy looks like this:

{
  "rest_dns_lookup": {
    "_source": {
      "dns": {
        "stats_info": {
          "alerts": [],
          "seen_by_isc": "top1m",
          "seen_by_you": "2021-04-24T22:13:48.000Z",
          "seen_by_web": "1994-12-28T05:00:00.000Z",
          "category": "ESTABLISHED",
          "age_at_first_seen": 9614.717916666666
        }
      }
    }
  }
}

The following filter

mutate {
  rename => {
    "[rest_dns_lookup][_source][dns][stats_info]" => "[dns][stats_info]"
  }
}

Causes errors that look like this:

[2022-02-08T14:36:00,271][ERROR][logstash.filters.ruby][main]
[32618881f49549c0470d25216965daa780c28aac4746edc4aa31daa5baee8ea4
]
 Ruby exception occurred: undefined method `-' for "2021-04-24T22:13:49.000Z":String {
    :class=>"NoMethodError",
    :backtrace=>[
        "(ruby filter code):2:in `block in filter_method'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'",
        "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'",
        "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'",
        "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'",
        "org/jruby/RubyArray.java:1821:in `each'",
        "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'",
        "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'",
        "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"
    ]
}

The weird part is, when I run the filter on a standalone logstash for testing, it works but when the config file is dropped in the conf.d folder of logstash running as a service, that's when this error accurs. I haven't been able to figure this one out. All I can see is that the error is referencing a "-"?

[Badger]

The exception is occurring in a ruby filter, not a mutate filter.

[Bryan_Hamilton]

Hi @Badger I thought it could be the case since it is working in standalone mode, but I have been going through all the subsequent filters in con.d directory and can't find which one has a filter that could cause this error. The Ruby error doesn't tell me much about the code in the file as some other logstash errors some times do. I am not much of a ruby expert so though I would post the error here in case somebody else can spot more informations from the errors message than I am able to. I will go though all the other config files again.

[Badger]

As I said, it is a ruby filter, so you are looking for

ruby {
    code => '
        ...
    '
}

Could be an attempt to manipulate a date.

input { generator { count => 1 lines => [ '' ] } }
filter {
    mutate { add_field => { "[foo]" => "2021-04-24T22:13:49.000Z" } }
    date { match => [ "foo", "ISO8601" ] target => "foo" }
    ruby { code => 'puts event.get("foo") - 10' }
}
output { stdout { codec => rubydebug { metadata => false } } }

prints 2021-04-24 18:13:39 -0400. If the date filter is commented out then it produces that ruby exception.

[Bryan_Hamilton]

Thanks for the hints @Badger! Guided by your hints, I was able to find the file with the problematic filter and exactly as you suspected it is manipulating the dates. We need to adjust the code to prevent the error from hapening but at least now I know exactly where the problematic code is. Thank you for your time you have been of great help!