Ignore wrong date value

Elastic Stack Logstash
1

Hi,

Hope I'm in the corrent sections. If not, please forgive.

My parsed log files contain some date fields, and in some cases, the filed is not filled with epoch time value but with an empty string or just an "0".

logstash-plain.log shows a warning,
[2017-06-01T13:09:16,477][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-1970.01.01", :_type=>"logs", :_routing=>nil}, 1970-01-01T00:00:00.000Z hostname %{message}], :response=>{"index"=>{"_index"=>"index-1970.01.01", "_type"=>"logs", "_id"=>"AVxjWGT4Ng_JYGwPfHYX", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [schedule_delivery_time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"000000000:\" is malformed at \"1380649:\""}}}}}

Can i use conditionals? If filed not in DATE format, to be just plain text or number?

Thanks.

2

I tried like that for example:

    if [retry_time] == "0" {
    mutate {
    convert => { "retry_time" => "integer" }
    }
    } else {
    date {
   match => ['retry_time', 'UNIX']
   target => 'retry_time'
   }
    }

But not sure if it's ok.

3

ES is complaining about the schedule_delivery_time field but you're mutating the retry_time field.

Please show an example of a failing event. You can use a stdout { codec => rubydebug } output to dump its raw contents.

4

i did the same for all time fields

    if [retry_time] == "0" {
    mutate {
    convert => { "retry_time" => "integer" }
    }
    } else {
    date {
   match => ['retry_time', 'UNIX']
   target => 'retry_time'
   }
    }

    if [schedule_delivery_time] == "0" {
    mutate {
    convert => { "schedule_delivery_time" => "integer" }
    }
    } else {
    date {
   match => ['schedule_delivery_time', 'UNIX']
   target => 'schedule_delivery_time'
   }
    }

    if [expiry_time] == "0" {
    mutate {
    convert => { "expiry_time" => "integer" }
    }
    } else {
    date {
    match => ['expiry_time', 'UNIX']
    target => 'expiry_time'
    }
    }

Output:
[2017-06-02T10:28:48,249][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:39:33.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn69aF9vceVn58Yf3E", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [retry_time] of different type, current_type [date], merged_type [long]"}}}}

Also can probably cause conflicts in diferent field types ?

5

if i remove index and reparse file got this errors:

[2017-06-02T10:35:14,499][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:46.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgRz", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T06:43:46.000Z\""}}}}}
[2017-06-02T10:35:14,499][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T06:43:49.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxn8bhk9vceVn58YgR2", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T06:43:48.000Z\""}}}}}

also mapping conflict

Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
Field Type Index Names
number mmsc-2017.06.02
date mmsc-2017.06.01, mmsc-2017.05.31, mmsc-2017.05.28, mmsc-2017.05.29, mmsc-2017.05.30

6

One problem at a time. Let's start with your initial question.

I repeat: Please show an example of a failing event. You can use a stdout { codec => rubydebug } output to dump its raw contents.

7

here are few examples:

[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:43.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKw", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:40:43.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:40:51.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhKy", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-05T07:40:50.000Z\""}}}}}
[2017-06-02T11:55:47,313][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mmsc-2017.06.02", :_type=>"logs", :_routing=>nil}, 2017-06-02T07:41:17.000Z localhost.localdomain %{message}], :response=>{"index"=>{"_index"=>"mmsc-2017.06.02", "_type"=>"logs", "_id"=>"AVxoO3jf9vceVn58YhK5", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [retry_time]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"2017-06-07T07:41:17.000Z\""}}}}}

and also a ok one:

{
           "mms_read_report" => "",
         "delivery_attempts" => "1",
              "o_error_type" => "46",
        "orig_content_sizes" => "",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T07:41:39.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602094139_115",
                 "thread_id" => "",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "16A78DC1",
        "dest_content_types" => "",
                  "queue_id" => "0000:16A78DC1:",
                       "apn" => "0000",
             "i_error_value" => "90",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "0000",
          "divert_from_addr" => "0000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "MMS_RR",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-07T07:41:38.000Z,
             "o_error_value" => "1",
                    "UAProf" => "HUAWEI-MHA-L29/Android-Mms/2.0",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "0000",
              "mms_msg_type" => "m-read-rec-ind",
                 "esm_class" => "0",
               "final_state" => "REJECTED",
       "registered_delivery" => "0",
                 "orig_idnt" => "0000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "1496389299:46.1/1.90",
              "cluster_name" => "text",
              "i_error_type" => "1",
        "orig_content_types" => "",
           "mms_charge_type" => "PREPAID",
                "@timestamp" => 2017-06-02T07:41:38.000Z,
                 "dest_locn" => "0000",
           "submission_time" => 2017-06-02T07:41:38.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "0000",
                 "orig_locn" => "0000",
               "storage_ref" => "27144024"
}
8

I don't want to see Logstash's error messages. I want to see the events themselves, like in your successful example. Comment out the elasticsearch output for now.

9

will send you a sample on private if possible

thanks

10

I only offer public help. You can obfuscate sensitive data if you wish.

11

here are some events:

{
           "mms_read_report" => "No",
         "delivery_attempts" => "1",
              "o_error_type" => "0",
        "orig_content_sizes" => "526|369573",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T06:40:41.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
                 "thread_id" => "2265a3b0",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "205DBD5B",
        "dest_content_types" => "",
                  "queue_id" => "000000:",
                       "apn" => "000000.000000.000000",
             "i_error_value" => "0",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "000000",
          "divert_from_addr" => "000000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "RECEIPT",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-09T06:40:41.000Z,
             "o_error_value" => "0",
                    "UAProf" => "SonyF5321/34.2.A.2.47",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "000000",
              "mms_msg_type" => "m-send-req",
                 "esm_class" => "1048834",
               "final_state" => "DELIVERED",
       "registered_delivery" => "1025",
                 "orig_idnt" => "000000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "",
              "cluster_name" => "MMSC-000000",
              "i_error_type" => "0",
        "orig_content_types" => "application/smil|image/jpeg",
           "mms_charge_type" => "POSTPAID",
                "@timestamp" => 2017-06-02T06:40:11.000Z,
                 "dest_locn" => "000000",
           "submission_time" => 2017-06-02T06:40:11.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "000000",
                 "orig_locn" => "000000",
               "storage_ref" => "24982325"
}
{
           "mms_read_report" => "",
         "delivery_attempts" => "1",
              "o_error_type" => "22",
        "orig_content_sizes" => "",
                    "oa_ton" => "1",
             "delivery_time" => 2017-06-02T06:40:41.000Z,
                      "type" => "logs",
                    "da_npi" => "1",
                      "path" => "/logs/mmsc/mmsc_elk_tcmmssim2_20170602084428_103",
                 "thread_id" => "",
                    "oa_npi" => "1",
                      "host" => "localhost.localdomain",
                    "msg_id" => "13B9CA51",
        "dest_content_types" => "",
                  "queue_id" => "000000:13B9CA51:",
                       "apn" => "000000.000000.000000",
             "i_error_value" => "40",
           "divert_from_ton" => "1",
                "retry_time" => 0,
               "protocol_id" => "0",
                  "priority" => "0",
                   "da_addr" => "000000",
          "divert_from_addr" => "000000",
               "data_coding" => "0",
                    "da_ton" => "1",
                  "msg_type" => "MMS_RR",
           "divert_from_npi" => "1",
               "expiry_time" => 2017-06-07T06:40:40.000Z,
             "o_error_value" => "5",
                    "UAProf" => "HUAWEI-VNS-L21/Android-Mms/2.0",
                   "charset" => "0",
                  "rcpt_num" => "1 of 1",
                 "dest_idnt" => "000000",
              "mms_msg_type" => "m-read-rec-ind",
                 "esm_class" => "0",
               "final_state" => "REJECTED",
       "registered_delivery" => "0",
                 "orig_idnt" => "000000",
        "pre_submission_len" => "0",
                  "@version" => "1",
        "dest_content_sizes" => "",
             "history_error" => "1496385641:22.5/1.40",
              "cluster_name" => "MMSC-000000",
              "i_error_type" => "1",
        "orig_content_types" => "",
           "mms_charge_type" => "PREPAID",
                "@timestamp" => 2017-06-02T06:40:40.000Z,
                 "dest_locn" => "000000",
           "submission_time" => 2017-06-02T06:40:40.000Z,
    "schedule_delivery_time" => 0,
                   "oa_addr" => "000000",
                 "orig_locn" => "000000",
               "storage_ref" => "27055026"
}
12

schedule_delivery_time contains the number zero, not the string "0". so your conditional needs to look like this:

if [schedule_delivery_time] == 0 {

Same thing with retry_time. However, expiry_time is of a timestamp type so it can't be compared to either 0 or "0".

13

it's that simple..

thank you Magnus for your support and sorry for bothering :slight_smile:

keep up the good work!

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.