_dateparsefailure

Elastic Stack Logstash
1

So I don't know why I got the so kind error :

_dateparsefailure

below You can find one of data input

|date|time|millisecond|

|18.11.2021|09:29:15|257|

code:

	ruby {
		code => "
			event_millisecond = ('000' + event.get('millisecond').to_s).split(//).last(3).join
			event.set('event_timestamp', event.get('date') + ' ' + event.get('time') + '.' + event_millisecond)
		"
	}

	date {
		match => ["event_timestamp", "yyyy-MM-dd H:m:s.SSS z"]
		timezone => "Europe/Berlin"
		target => "@timestamp"
	}

and output:

[...]
            "@timestamp" => 2022-01-20T21:45:31.562Z,
             "ebm_processing_time" => 2022-01-20T21:45:32.404Z,
               
                     "tags" => [
        [0] "_dateparsefailure"
    ]

Thanks for the tips

2

What do the [date], [time], [millisecond] and [event_timestamp] fields look like in the rubydebug output?

3

these fields are a part of CSV input file of column:

columns => ["event_id","event_result","date","time","millisecond","duration" .....

so at end of code I've removed that fields

mutate {
			remove_field => "date",  "millisecond", "time"

except event_millisecond field
so it could be a problem? I don't think so,

4 
[2022-01-20T20:51:51,445][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "\n\t\t\tevent_millisecond = ('000' + event.get('millisecond').to_s).split(//).last(3).join\n\t\t\tevent.set('event_timestamp', event.get('date') + ' ' + event.get('time') + '.' + event_millisecond)\n\t\t"
[2022-01-20T20:51:51,506][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@match = ["event_timestamp", "yyyy-MM-dd H:m:s.SSS z"]
[2022-01-20T20:52:31,396][DEBUG][logstash.filters.mutate  ][main][c9428ede1d3c0eaf9f91e05fa04540a1ceee1a924fb30a8b47bfc58857444bcb] filters/LogStash::Filters::Mutate: removing field {:field=>"event_timestamp"}
5

That is not what your timestamp looks like, so it is not going to match. Try "dd.MM.yyyy H:m:s.SSS"

6

hmm I'm afraid that is not the reason I've changed it already this format for timestamp

	Line 426: [2022-01-20T22:55:58,479][DEBUG][logstash.filters.csv     ] config LogStash::Filters::CSV/@columns = ["event_id", "event_result", "date", "time", "millisecond", "duration", "a21_message_type", "a_msisdn", "access_type", "activation_trigger", "activation_type", "active_timer", "age_of_location_estimate", "amf_ue_ngap_id", "amfi", "apn", "attach_type", "back_off_timer", "bearers", "bearers_2", "cause_code", "cause_code2", "cause_prot_type", "cdma2000_type", "cell_access_mode", "ci", "ciot", "combined_tau_type", "connection_suspend_and_resume", "coverage_level", "cs_fallback_service_type", "csfb_triggered", "csg_id", "csg_membership_status", "deactivation_trigger", "deconnect_pdn_type", "decor_info", "default_bearer_id", "deregistration_trigger", "detach_trigger", "detach_type", "dnn", "dnn_selection_mode", "donas", "dropped_pdp", "duration", "duration_psm", "eci", "edrx_delay_paging_time", "edrx_info2_lte", "edrx_info_gsm", "edrx_info_lte", "edrx_ptw", "edrx_tedrx", "emergency", "emergency_registration", "enb_list_length", "enb_list_length_ext", "enb_serial_number", "enb_sn", "enb_sn_ext", "enb_ue_s1ap_id", "eps_fallback_triggered", "esm_dummy_msg_in_req", "esmlc", "event_notification_cause", "forwarded_5g_bearer_to_3g", "gad_shape", "ggsn", "gmlc", "gnodeb_ue_ngap_id", "gtp_piggybacked", "gw_from_sapc", "handoff_status", "handover_node_role", "handover_rat_change_type", "handover_sgw_change_type", "handover_type", "hlr", "homezone_identity", "http_problem_details_cause", "imeisv", "imsi", "intra_rau_type", "is_imeisv_changed", "is_pei_changed", "ismf_fqdn", "ismf_instance", "iws_address", "l_cause_prot_type", "l_paging_trigger", "l_rat", "l_rat_type", "l_service_req_trigger", "l_service_req_trigger2", "last_mcc", "last_mnc", "lcs_client_name", "lcs_client_type", "lcs_reference_number", "linked_nsapi", "location", "location_event", "location_reporting_delayed", "lte_m_ue_indication", "mico_mode", "mme_ue_s1ap_id", "mmei", "modification_trigger", "ms_address", "ms_requested ...
	Line 440: [2022-01-20T22:55:58,512][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "\n\t\t\tevent_millisecond = ('000' + event.get('millisecond').to_s).split(//).last(3).join\n\t\t\tevent.set('event_timestamp', event.get('date') + ' ' + event.get('time') + '.' + event_millisecond)\n\t\t"
	Line 451: [2022-01-20T22:55:58,525][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"date", :type=>"filter", :class=>LogStash::Filters::Date}
	Line 451: [2022-01-20T22:55:58,525][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"date", :type=>"filter", :class=>LogStash::Filters::Date}
	Line 452: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@match = ["event_timestamp", "yyyy-MM-dd H:m:s.SSS"]
	Line 452: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@match = ["event_timestamp", "yyyy-MM-dd H:m:s.SSS"]
	Line 453: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@id = "03b1e54f733946639884a41381e998dbe55d8e97d4e73eee4c392a91be9734e7"
	Line 453: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@id = "03b1e54f733946639884a41381e998dbe55d8e97d4e73eee4c392a91be9734e7"
	Line 454: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@timezone = "Europe/Berlin"
	Line 454: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@timezone = "Europe/Berlin"
	Line 455: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@target = "@timestamp"
	Line 455: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@target = "@timestamp"
	Line 456: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@enable_metric = true
	Line 456: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@enable_metric = true
	Line 457: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@add_tag = []
	Line 457: [2022-01-20T22:55:58,544][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@add_tag = []
	Line 458: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@remove_tag = []
	Line 458: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@remove_tag = []
	Line 459: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@add_field = {}
	Line 459: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@add_field = {}
	Line 460: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@remove_field = []
	Line 460: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@remove_field = []
	Line 461: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@periodic_flush = false
	Line 461: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@periodic_flush = false
	Line 462: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"]
	Line 462: [2022-01-20T22:55:58,545][DEBUG][logstash.filters.date    ] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsefailure"]
	Line 463: [2022-01-20T22:55:58,651][DEBUG][org.logstash.filters.DateFilter] Date filter with format=yyyy-MM-dd H:m:s.SSS, locale=null, timezone=Europe/Berlin built as org.logstash.filters.parser.JodaParser
	Line 472: [2022-01-20T22:55:58,689][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@init = "['date', 'tzinfo'].each(&method(:require))"
	Line 1940: [2022-01-20T22:56:34,333][DEBUG][logstash.filters.mutate  ][main][e92654142c0004e6c1dee980b30e75daede5a9bee96302882541976c8ed7b0a1] filters/LogStash::Filters::Mutate: removing field {:field=>"date"}
7

Also I'm checking on logstash 7.16.3

8

OK, I am not going to waste my time pointing out problems with setups that you now say you are not using.

9

What I've meant only it was that "dd.MM.yyyy H:m:s.SSS" format also impacted with the same failure

10

so for fix that issue I put

date {
match => ["event_timestamp", "dd-MM-yyyy H:m:s.SSS", "yyyy-MM-dd H:m:s.SSS", "ISO8601"]

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.