Failed to parse date field

Elastic Stack Logstash
1

I want to send Logs from Winlogbeat, via Logstash into Elasticsearch.
From Winlogbeat directly to Elasticsearch is it working.
However if I put the output to Logstash, I am getting this warning, that the date field could not be parsed.

[2022-04-12T16:29:46,232][WARN ][logstash.outputs.elasticsearch][main][c73f613a57c5bc86841151419c201d766d4997a1bd86a20104326cc4fceca942] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-8.0.1-2022.04.12", :routing=>nil}, {"log"=>{"level"=>"information"}, "message"=>"Engine state is changed from None to Available. \n\nDetails: \n\tNewEngineState=Available\n\tPreviousEngineState=None\n\n\tSequenceNumber=77397\n\n\tHostName=OpsMgr PowerShell Host\n\tHostVersion=7.0.5000.0\n\tHostId=NUMBERID\n\tHostApplication=C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe -Embedding\n\tEngineVersion=5.1.17763.2268\n\tRunspaceId=RUNSPACEID\n\tPipelineId=\n\tCommandName=\n\tCommandType=\n\tScriptName=\n\tCommandPath=\n\tCommandLine=", "host"=>{"name"=>"HOSTADDRESS", "os"=>{"name"=>"Windows Server 2019 Standard", "kernel"=>"10.0.17763.2686 (WinBuild.160101.0800)", "family"=>"windows", "build"=>"17763.2686", "version"=>"10.0", "type"=>"windows", "platform"=>"windows"}, "ip"=>["IP"], "id"=>"ID", "mac"=>["MAC"], "hostname"=>"HOSTNAME", "architecture"=>"x86_64"}, "event"=>{"created"=>"2022-04-12T14:29:45.065Z", "provider"=>"PowerShell", "original"=>"Engine state is changed from None to Available. \n\nDetails: \n\tNewEngineState=Available\n\tPreviousEngineState=None\n\n\tSequenceNumber=77397\n\n\tHostName=OpsMgr PowerShell Host\n\tHostVersion=7.0.5000.0\n\tHostId=HOSTID\n\tHostApplication=C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe -Embedding\n\tEngineVersion=5.1.17763.2268\n\tRunspaceId=RUNSPACEID\n\tPipelineId=\n\tCommandName=\n\tCommandType=\n\tScriptName=\n\tCommandPath=\n\tCommandLine=", "kind"=>"event", "action"=>"Engine Lifecycle", "code"=>"400"}, "@version"=>"1", "tags"=>["beats_input_codec_plain_applied"], "@timestamp"=>2022-04-12T14:29:45.011Z, "agent"=>{"name"=>"HOSTNAME", "id"=>"ID", "version"=>"8.0.1", "type"=>"winlogbeat", "ephemeral_id"=>"97bdcf5c-a533-4e34-8748-dcb7c069b66a"}, "winlog"=>{"provider_name"=>"PowerShell", "event_id"=>"400", "opcode"=>"Info", "computer_name"=>"NAME", "task"=>"Engine Lifecycle", "api"=>"wineventlog", "record_id"=>358936, "channel"=>"Windows PowerShell", "keywords"=>["Classic"], "event_data"=>{"param1"=>"Available", "param2"=>"None", "param3"=>"\tNewEngineState=Available\n\tPreviousEngineState=None\n\n\tSequenceNumber=77397\n\n\tHostName=OpsMgr PowerShell Host\n\tHostVersion=7.0.5000.0\n\tHostId=HOSTID\n\tHostApplication=C:\\Program Files\\Microsoft Monitoring Agent\\Agent\\MonitoringHost.exe -Embedding\n\tEngineVersion=5.1.17763.2268\n\tRunspaceId=RUNSPACEID\n\tPipelineId=\n\tCommandName=\n\tCommandType=\n\tScriptName=\n\tCommandPath=\n\tCommandLine="}}, "ecs"=>{"version"=>"8.0.0"}}], :response=>{"index"=>{"_index"=>"winlogbeat-8.0.1-2022.04.12", "_id"=>"ElIuHoABphNDltYONN9U", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [winlog.event_data.param1] of type [date] in document with id 'ElIuHoABphNDltYONN9U'. Preview of field's value: 'Available'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [Available] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}

My Logstash Conf is very basic right now. (Filebeat is working with this)

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
  beats {
    port => 5044
  }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    ssl => true
    cacert => 'C:\path to cert\logstash-8.0.1\config\certs\http_ca.crt'
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => elastic
    password => password
  }
}

Any ideas? What am I missing?

2

Most likely you have dynamic mapping enabled in elasticsearch. If the first document indexed has a value of [winlog][event_data][param1] that looks like a date then elasticsearch will set the field type to date, and any document on which that field is not a valid date will get that mapping exception. That is true no matter whether the event comes from filebeat or logstash. It just depends what the first document indexed looks like.

You need to add a template or otherwise set the mapping on the index. Note that you cannot change the type of a field once created. You will need to create a new index.

See this post for more colour.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.