Ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:35.919"

Elastic Stack Logstash
1

I am trying to find a diff betweentwo dates.
I have wrote the following con file which use Ruby in order to caculate the date diff.
I recived : ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:35.919"

The conf file:
input {
file {
path => "/home/logstash/cdr_files/top10.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
        csv {
             separator => ","
           columns => [
                       "P9","P11","P21","P30"
                      ]
             }

        mutate { rename => { "P9" => "CALLING_NUMBER" } }
        mutate { rename => { "P11" => "CALLED_NUMBER" } }
        mutate { rename => { "P21" => "ING_SIGNAL_START_TIME" } }
        mutate { rename => { "P30" => "ING_CALL_ANSWER_TIME" } }

        date {
                match => [ "P21", "YYYY-MM-dd'+'HH:mm:ss.SSS" ]
                target => [ "ING_SIGNAL_START_TIME" ]
                remove_field => [ "P21" ]
              }


        date {
                match => [ "P30", "YYYY-MM-dd'+'HH:mm:ss.SSS" ]
                target => [ "ING_CALL_ANSWER_TIME" ]
                remove_field => [ "P30" ]
              }

ruby {
    init => "require 'time'"
    code => "
           starttime = Time.iso8601(event.get('ING_SIGNAL_START_TIME').to_s).to_f;
           endtime   = Time.iso8601(event.get('ING_ADDRESS_COMPLETE_TIME').to_s).to_f;
           event.set('time_diff', endtime - starttime);
            "
    add_tag => [ "ING_SST_MINUS_ING_ACT_CALCUALTED" ]
}

}

output {
        elasticsearch {
                        hosts => "localhost"
                        index => "top10"
                        document_type => "quick_log"
                      }
        stdout {}

The error

[INFO ] 2019-05-28 12:24:10.754 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2019-05-28 12:24:11.498 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[ERROR] 2019-05-28 12:24:12.844 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:35.919"
[ERROR] 2019-05-28 12:24:12.849 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:10.946"
[ERROR] 2019-05-28 12:24:12.850 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:39.416"
[ERROR] 2019-05-28 12:24:12.850 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:36.915"
[ERROR] 2019-05-28 12:24:12.851 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:38.636"
[ERROR] 2019-05-28 12:24:12.851 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:22:58.574"
[ERROR] 2019-05-28 12:24:12.851 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:20:24.000"
[ERROR] 2019-05-28 12:24:12.852 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:00.175"
[ERROR] 2019-05-28 12:24:12.852 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:38.324"
[ERROR] 2019-05-28 12:24:12.853 [[main]>worker1] ruby - Ruby exception occurred: invalid date: "2019-04-18+06:23:40.887"
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
     "ING_CALL_ANSWER_TIME" => nil,
                     "tags" => [
        [0] "_rubyexception"
    ],
            "CALLED_NUMBER" => "8656",
                  "message" => "1679,8656,2019-04-18+06:23:35.919,\r",
    "ING_SIGNAL_START_TIME" => "2019-04-18+06:23:35.919",
                 "@version" => "1",
           "CALLING_NUMBER" => "1679",
                     "host" => "elk7-lab",
               "@timestamp" => 2019-05-28T09:24:12.171Z,
                     "path" => "/home/logstash/cdr_files/top10.csv"
}
{
     "ING_CALL_ANSWER_TIME" => nil,
                     "tags" => [
        [0] "_rubyexception"
    ],
            "CALLED_NUMBER" => "0878",
                  "message" => "9.23E+11,0878,2019-04-18+06:23:10.946,\r",
    "ING_SIGNAL_START_TIME" => "2019-04-18+06:23:10.946",
                 "@version" => "1",
           "CALLING_NUMBER" => "9.23E+11",
                     "host" => "elk7-lab",
               "@timestamp" => 2019-05-28T09:24:12.217Z,
                     "path" => "/home/logstash/cdr_files/top10.csv"

Please advise whats wrong

Thanks Alot

2

You rename P21 and P30, then after that you try to parse those fields, which no longer exist. And the ruby filter references a non-existant ING_ADDRESS_COMPLETE_TIME field. How about

ruby {
    init => "require 'time'"
    code => "
       starttime = event.get('ING_SIGNAL_START_TIME').to_f;
       endtime   = event.get('ING_CALL_ANSWER_TIME').to_f;
       event.set('time_diff', endtime - starttime);
    "
    add_tag => [ "ING_SST_MINUS_ING_ACT_CALCUALTED" ]
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.