Ignore wrong date value

mitabrev · June 1, 2017, 11:46am

Hi,

Hope I'm in the corrent sections. If not, please forgive.

My parsed log files contain some date fields, and in some cases, the filed is not filled with epoch time value but with an empty string or just an "0".

logstash-plain.log shows a warning,
[2017-06-01T13:09:16,477][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-1970.01.01", :_type=>"logs", :_routing=>nil}, 1970-01-01T00:00:00.000Z hostname %{message}], :response=>{"index"=>{"_index"=>"index-1970.01.01", "_type"=>"logs", "_id"=>"AVxjWGT4Ng_JYGwPfHYX", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [schedule_delivery_time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"000000000:\" is malformed at \"1380649:\""}}}}}

Can i use conditionals? If filed not in DATE format, to be just plain text or number?

Thanks.

mitabrev · June 2, 2017, 8:16am

I tried like that for example:

    if [retry_time] == "0" {
    mutate {
    convert => { "retry_time" => "integer" }
    }
    } else {
    date {
   match => ['retry_time', 'UNIX']
   target => 'retry_time'
   }
    }

But not sure if it's ok.

magnusbaeck · June 2, 2017, 8:22am

ES is complaining about the schedule_delivery_time field but you're mutating the retry_time field.

Please show an example of a failing event. You can use a stdout { codec => rubydebug } output to dump its raw contents.

mitabrev · June 2, 2017, 8:33am

i did the same for all time fields

    if [retry_time] == "0" {
    mutate {
    convert => { "retry_time" => "integer" }
    }
    } else {
    date {
   match => ['retry_time', 'UNIX']
   target => 'retry_time'
   }
    }

    if [schedule_delivery_time] == "0" {
    mutate {
    convert => { "schedule_delivery_time" => "integer" }
    }
    } else {
    date {
   match => ['schedule_delivery_time', 'UNIX']
   target => 'schedule_delivery_time'
   }
    }

    if [expiry_time] == "0" {
    mutate {
    convert => { "expiry_time" => "integer" }
    }
    } else {
    date {
    match => ['expiry_time', 'UNIX']
    target => 'expiry_time'
    }
    }

Output:
[2017-06-02T10:28:48,249][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:39:33.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn69aF9vceVn58Yf3E", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [retry_time] of different type, current_type [date], merged_type [long]"}}}}

Also can probably cause conflicts in diferent field types ?

mitabrev · June 2, 2017, 8:40am

if i remove index and reparse file got this errors:

[2017-06-02T10:35:14,499][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:46.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgRz", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T06:43:46.000Z\""}}}}}
[2017-06-02T10:35:14,499][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:49.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgR2", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T06:43:48.000Z\""}}}}}

also mapping conflict

Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
Field Type Index Names
number mmsc-2017.06.02
date mmsc-2017.06.01, mmsc-2017.05.31, mmsc-2017.05.28, mmsc-2017.05.29, mmsc-2017.05.30

magnusbaeck · June 2, 2017, 9:03am

One problem at a time. Let's start with your initial question.

I repeat: Please show an example of a failing event. You can use a stdout { codec => rubydebug } output to dump its raw contents.

mitabrev · June 2, 2017, 10:06am

here are few examples:

[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:43.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKw", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:40:43.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:51.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKy", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T07:40:50.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:41:17.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhK5", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:41:17.000Z\""}}}}}

and also a ok one:

{
           "mms_read_report" => "",
         "delivery_attempts" => "1",
              "o_error_type" => "46",
        "orig_content_sizes" => "",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T07:41:39.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602094139_115",
                 "thread_id" => "",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "16A78DC1",
        "dest_content_types" => "",
                  "queue_id" => "0000:16A78DC1:",
                       "apn" => "0000",
             "i_error_value" => "90",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "0000",
          "divert_from_addr" => "0000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "MMS_RR",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-07T07:41:38.000Z,
             "o_error_value" => "1",
                    "UAProf" => "HUAWEI-MHA-L29/Android-Mms/2.0",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "0000",
              "mms_msg_type" => "m-read-rec-ind",
                 "esm_class" => "0",
               "final_state" => "REJECTED",
       "registered_delivery" => "0",
                 "orig_idnt" => "0000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "1496389299:46.1/1.90",
              "cluster_name" => "text",
              "i_error_type" => "1",
        "orig_content_types" => "",
           "mms_charge_type" => "PREPAID",
                "@timestamp" => 2017-06-02T07:41:38.000Z,
                 "dest_locn" => "0000",
           "submission_time" => 2017-06-02T07:41:38.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "0000",
                 "orig_locn" => "0000",
               "storage_ref" => "27144024"
}

magnusbaeck · June 2, 2017, 11:23am

I don't want to see Logstash's error messages. I want to see the events themselves, like in your successful example. Comment out the elasticsearch output for now.

mitabrev · June 2, 2017, 11:59am

will send you a sample on private if possible

thanks

magnusbaeck · June 2, 2017, 12:07pm

I only offer public help. You can obfuscate sensitive data if you wish.

mitabrev · June 2, 2017, 12:14pm

here are some events:

{
           "mms_read_report" => "No",
         "delivery_attempts" => "1",
              "o_error_type" => "0",
        "orig_content_sizes" => "526|369573",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T06:40:41.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
                 "thread_id" => "2265a3b0",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "205DBD5B",
        "dest_content_types" => "",
                  "queue_id" => "000000:",
                       "apn" => "000000.000000.000000",
             "i_error_value" => "0",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "000000",
          "divert_from_addr" => "000000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "RECEIPT",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-09T06:40:41.000Z,
             "o_error_value" => "0",
                    "UAProf" => "SonyF5321/34.2.A.2.47",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "000000",
              "mms_msg_type" => "m-send-req",
                 "esm_class" => "1048834",
               "final_state" => "DELIVERED",
       "registered_delivery" => "1025",
                 "orig_idnt" => "000000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "",
              "cluster_name" => "MMSC-000000",
              "i_error_type" => "0",
        "orig_content_types" => "application/smil|image/jpeg",
           "mms_charge_type" => "POSTPAID",
                "@timestamp" => 2017-06-02T06:40:11.000Z,
                 "dest_locn" => "000000",
           "submission_time" => 2017-06-02T06:40:11.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "000000",
                 "orig_locn" => "000000",
               "storage_ref" => "24982325"
}
{
           "mms_read_report" => "",
         "delivery_attempts" => "1",
              "o_error_type" => "22",
        "orig_content_sizes" => "",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T06:40:41.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
                 "thread_id" => "",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "13B9CA51",
        "dest_content_types" => "",
                  "queue_id" => "000000:13B9CA51:",
                       "apn" => "000000.000000.000000",
             "i_error_value" => "40",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "000000",
          "divert_from_addr" => "000000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "MMS_RR",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-07T06:40:40.000Z,
             "o_error_value" => "5",
                    "UAProf" => "HUAWEI-VNS-L21/Android-Mms/2.0",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "000000",
              "mms_msg_type" => "m-read-rec-ind",
                 "esm_class" => "0",
               "final_state" => "REJECTED",
       "registered_delivery" => "0",
                 "orig_idnt" => "000000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "1496385641:22.5/1.40",
              "cluster_name" => "MMSC-000000",
              "i_error_type" => "1",
        "orig_content_types" => "",
           "mms_charge_type" => "PREPAID",
                "@timestamp" => 2017-06-02T06:40:40.000Z,
                 "dest_locn" => "000000",
           "submission_time" => 2017-06-02T06:40:40.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "000000",
                 "orig_locn" => "000000",
               "storage_ref" => "27055026"
}

magnusbaeck · June 2, 2017, 1:01pm

schedule_delivery_time contains the number zero, not the string "0". so your conditional needs to look like this:

if [schedule_delivery_time] == 0 {

Same thing with retry_time. However, expiry_time is of a timestamp type so it can't be compared to either 0 or "0".

mitabrev · June 2, 2017, 3:11pm

it's that simple..

thank you Magnus for your support and sorry for bothering

keep up the good work!

system · June 30, 2017, 3:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Conditional for empty field Logstash	3	23129	July 4, 2017
Date mapping issue Logstash	17	3380	January 19, 2018
Logstash trying to parse date to the wrong field Logstash	4	1097	August 9, 2017
Logstash time parsing error Logstash	4	1629	July 6, 2017
Date parsing errors I need help to debug Logstash	5	958	July 2, 2020

Ignore wrong date value

Related topics