Logstash doesn't parse date

Elastic Stack Logstash
1

Hello!
I created config in previous version of logstash and it was working good.
After logstash update (removed and install new version) i used settings from my config, but it doesn't parse date.
Example xml is for section in condition "else"
In example xml error fileds is TimeStamp, StartDate, EndDate, LastApp_date.
Logstash puting _dateparsefailure tag and provide this fields as string. I expect it will be date fields, EndDate will be @timestamp.

Here is my config

filter
{
        if [fields][stats_file] == "true"
        {
                grok
                {
                        match => { "message" => "\[%{DATA}\]\[%{DATA:date}\]\[%{DATA}\]\[%{DATA}\] (them\=%{DATA:them}\:)?(time\=%{NUMBER:time_stamp}\:)?(Workbench\=%{DATA:Workbench}\:)?(Appname\=%{DATA:AppName}\:)?(Brand\=%{DATA:Brand}\:)?(NLS\=%{DATA:NLS}\:)?(elps\=%{NUMBER:elps}\:)?(user\=%{NUMBER:UserNumber}\:)?(host\=%{HOSTNAME:Host}\:)?(upid\=%{NUMBER:upid}\:)?(tenant\=%{DATA:Tenant}\:)?(SessionID\=%{DATA:SessionID}\:)?(UserName\=%{DATA:UserName}\:)?(Duration\=%{NUMBER:Duration})?" }
                }
                date
                {
                        match => [ "[date]", "yyyy-MM-dd@HH:mm:ss.SSS" ]
                }
                mutate
                {
                        remove_field => [ "date", "message", "[fields][stats_file]", "[prospector][type]", "[input][type]" ]
                        convert => { "Duration" => "integer" }
                }
        }
        else if [fields][abend_file] == "true"
        {
                xml
                {
                        source => "message"
                        xpath =>
                        [
                                "/root/Status/text()", "Status",
                                "/root/SessionID/text()", "SessionID"
                        ]
                        store_xml => true
                        target => "oic"
                        remove_field => [ "message" ]
                }
                date
                {
                        match => [ "oic.EndDate", "ISO8601", "d-M-yyyy HH:mm:ss", "dd.MM.yyyy HH:mm:ss", "d.M.yyyy H:m:s", "ss.MM.yyyy HH:mm:ss", "yyyy/MM/dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.ZZZ" ]
                }
        }
        else if [fields][report_file] == "true"
        {
                xml
                {
                        source => "message"
                        store_xml => true
                        target => "oic"
                        remove_field => [ "message" ]
                }
                date
                {
                        match => [ "[oic][Date]", "dd.MM.yyyy HH:mm:ss", "d.M.yyyy H:m:s" ]
                }
        }
        else
        {
                xml
                {
                        source => "message"
                        xpath =>
                        [
                                "/root/Status/text()", "Status",
                                "/root/SessionID/text()", "SessionID"
                        ]
                        store_xml => true
                        target => "oic"
                        remove_field => [ "message" ]
                }
                date
                {
                        match => [ "[oic][EndDate]", "d.M.YYYY H:m:s", "dd.MM.YYYY HH:mm:ss" ]
                }
                date
                {
                        match => [ "[oic][TimeStamp]", "dd-MM-YYYY HH:mm:ss", "d-M-YYYY H:m:s" ]
                        target => [ "[oic][TimeStamp]" ]
                }
                date
                {
                        match => [ "[oic][StartDate]", "d.M.YYYY H:m:s", "dd.MM.YYYY HH:mm:ss" ]
                        target => [ "[oic][StartDate]" ]
                }
                date
                {
                        match => [ "[oic][LastApp_date]", "dd-MM-YYYY HH:mm:ss", "d-M-YYYY H:m:s" ]
                        target => [ "[oic][LastApp_date]" ]
                }
                mutate
                {
                        convert => { "[oic][duration]" => "integer" }
                        convert => { "[oic][BuildHF]" => "integer"  }
                }
        }
}

And here is xml example, what i'm parsing:
< root >
.
.
.
< TimeStamp >16-10-2018 10:38:36< TimeStamp >
< StartDate >16.10.2018 10:38:36< /StartDate >
< EndDate >16.10.2018 11:40:55< /EndDate >
<LastApp_date>2018-10-16 14:40:50</LastApp_date>
.
.
.
< /root >

2

Hello, could you please properly format your code to make it easier to read! Use the "Preformatted Text" option. Can you also explain what the incorrect field is showing and how it is supposed to look?

3

Hello!
Field [oic][Date] have target @timestamp, but @timestamp = "date of log upload"
Other fields with specified target looks as date, but have type string.

Now i added to every date section option "tag_on_failure => [ "TARGET_NAME_ERROR" ]", and i have this tag in kibana. Some mistake in config, i think, but can't find it. I checked documentation, but didn't find any differences between old version and new.
Clean installation didn't resolved problem.

Server installed on cent os 7 minimal
filebeat client on Ubuntu 16.04.3 LTS

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.